Method and apparatus for automatic linear object identification using identified terrain types in images

ABSTRACT

A method and apparatus are provided for identifying linear objects in an image. Terrain types in the image are identified, and a gradient vector image, which identifies a gradient magnitude value and a gradient direction value for each pixel of the image, is generated from the image. Lines in the gradient vector image are identified using the identified terrain types in each portion of the image. It is determined whether the identified lines are perpendicular, collinear, or parallel to another line among the identified lines in the gradient vector image. Lines among the identified line are eliminated which are determined to not be perpendicular, collinear, or parallel to another line among the identified lines in the gradient vector image. Linear objects are identified using the remaining identified lines which have not been eliminated.

CROSS REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. application Ser. No.10/622,144, filed Jul. 18, 2003, now pending.

BACKGROUND

1. Field of Invention

The present invention relates to processing of image data. Moreparticularly, the present invention relates to methods and apparatus foridentifying objects in images.

2. Background Information

Historically, reconnaissance information has provided importantinformation used in planning military operations. For example, prior tothe advent of photography, scouts would be sent out to collectinformation regarding natural resources such as lakes and rivers, enemytroop information and the like. With the advent of photography, thesescouts would provide reconnaissance information by capturing a scene ofenemy installations, battlefields, and the like, using photographs. Astechnology advances, new methods are provided for collectingreconnaissance information. For example, it is quite common today tohave reconnaissance planes, manned or remotely controlled, or satellitescapture a scene for reconnaissance purposes. In addition to conventionalphotographic techniques, a scene can be captured using infrareddetectors and the like.

Typically scenes captured by reconnaissance techniques have beenanalyzed by humans in order to determine the content of the capturedscene. For example, a human would analyze a photograph to determine thelocation of bodies of water, the location of enemy troops and thelocation of man-made objects such as buildings and lines ofcommunication. The human who analyzed the photograph would then have torelay the determined information to people in the field, for example, toan airplane pilot in order to identify targets. However, using humans toanalyze photographs is very labor intensive. Further, there can be aconsiderable delay between the time when a scene is captured and thetime in which the information in the captured scene is relayed topersons in the field.

SUMMARY OF THE INVENTION

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying objects in an image. Inaccordance with this embodiment, an image with a first resolution isreceived. The image is processed at a second resolution to identify anobject. The image is processed at the first resolution using theidentified object to identify another object, wherein the firstresolution is higher than the second resolution.

In accordance with one embodiment of the present invention, a method andapparatus are provided for automatically identifying objects in animage. An image is received. A second image is generated identifyingareas of the image which border regions of different intensities. Athird image is generated identifying portions of the image for which anaverage gradient magnitude of the portion is greater than a threshold.The second image is processed to produce a fourth image, the fourthimage identifying lines in the image. The image is segmented into aplurality of regions. It is determined which of the plurality of regionsis a background region not containing said objects. Adjacent regionswhich are not background regions are merged. Objects in the mergedadjacent regions are identified.

In accordance with one embodiment of the present invention, a method andapparatus are provided for automatically identifying bodies of water inan image. A first image at a first resolution is received. Said image ata second resolution is processed to produce a second image identifyingbodies of water in the image at said second resolution. Said image isprocessed at a third resolution to produce a third image identifyingbodies of water in the image at said third resolution. Bodies of waterare automatically identified in the first image using said second andthird image.

In accordance with one embodiment of the present invention, a method andapparatus are provided for automatically identifying objects in animage. Terrain types in the image are identified. A second image isgenerated identifying areas of the image which border regions ofdifferent intensities by identifying a gradient magnitude value for eachpixel of the image. A filtered image is generated from the second image,the filtered image identifying potential objects which have a smallerradius than the size of a filter and a different brightness thanbackground pixels surrounding the potential objects. The second imageand the filtered image are compared to identify potential objects as anobject, a potential object is identified as an object if the potentialobject has a gradient magnitude greater than a threshold gradientmagnitude, and the threshold gradient magnitude is based on the terraintype identified in the portion of the image where the potential objectis located.

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying linear objects in an image.Terrain types in the image are identified. A gradient vector image isgenerated from the image, the gradient vector image identifying agradient magnitude value and a gradient direction value for each pixelof the image. Lines in the gradient vector image are identified usingthe identified terrain types in each portion of the image. It isdetermined whether the identified lines are perpendicular, collinear, orparallel. Lines which are not perpendicular, collinear, or parallel withanother line in the gradient vector image are eliminated. Linear objectsare identified using the remaining lines.

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying objects in an image. A gradientvector image is generated from the image, the gradient vector imageidentifying a gradient magnitude value and a gradient direction for eachpixel of the image. Lines in the gradient vector image are identified.It is determined whether the identified lines are perpendicular. It isdetermined whether more than a predetermined number of pixels on each ofthe lines identified as perpendicular have a gradient magnitude greaterthan a predetermined threshold. It is determined whether the individuallines which are identified as perpendicular are within a predetermineddistance of each other. A portion of the image is identified as anobject if the identified lines are perpendicular, more than thepredetermined number of pixels on each of the lines have a gradientmagnitude greater than the predetermined threshold, and are within apredetermined distance of each other.

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying linear objects in an image. Animage with a first resolution is received. A filtered image is generatedfrom the image, the filtered image identifying potential objects whichhave a smaller radius than the size of a filter and a differentbrightness than pixels surrounding the potential objects. A second imageis received identifying regions in the image with the first resolutionwhich are not to be processed. A third image is generated by removingregions in the filtered image which are identified in the another imageas regions in the image which are not to be processed. Lines areidentified in the third image. A fourth image is generated by removinglines identified in the third image which do not meet predeterminedcriteria. Linear objects are identified in the image using the remaininglines in the fourth image.

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying linear objects in an image. Animage with a first resolution is received. The image is processed toproduce an image at a second resolution. A filtered image is generatedfrom the image at the second resolution. A second image identifyingportions of the image with the first resolution which are not to beprocessed is received. A third image is generated by removing portionsof the filtered image which are identified in the second image asportions of the image which are not to be processed. Lines in the thirdimage are identified. A fourth image is generated by removing linesidentified in the third image which do not meet predetermined criteria.Linear objects in the image are identified using the remaining lines inthe fourth image.

In accordance with one embodiment of the present invention, a method andapparatus are provided for identifying linear objects in an image. Afirst and second image identifying linear objects are received, thefirst image having a first resolution and the second image having asecond resolution. The first and second images are processed to producea third image, wherein the processing combines linear objects from thefirst and second image. Linear objects in the image are identified usingthe third image.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects and advantages of the invention will become apparent tothose skilled in the art upon reading the following detailed descriptionof preferred embodiments, in conjunction with the accompanying drawings,wherein like reference numerals have been used to designate likeelements, and wherein:

FIG. 1 is a block diagram of a system for processing image data inaccordance with exemplary embodiments of the present invention.

FIG. 2 is a block diagram of the various functional processing blocks ofthe image processing block of FIG. 1.

FIG. 3 illustrates matrices employed for the determination of gray levelcocurrence.

FIG. 4 illustrates processing performed to identify terrain employinggray level cocurrence processing in accordance with exemplaryembodiments of the present invention.

FIG. 5 illustrates the calculation of various features employed in thegray level cocurrence processing.

FIG. 6 is an overview of the various processing employed by the cloudmask processing and activity detection blocks of FIG. 2.

FIGS. 7A-7C illustrate an exemplary technique for downsampling an imagewith a first resolution.

FIG. 8 illustrates an overview of the processing performed by theprimitive extraction processing block of FIG. 6.

FIGS. 9A and 9B illustrate exemplary Sobel templates in accordance withthe present invention.

FIG. 10 illustrates the four templates employed for edge thinningprocessing.

FIGS. 11A and 11B illustrate a line segment before and after edgethinning.

FIG. 12 illustrates the patterns employed to remove the 4-connectedpixels.

FIGS. 13A and 13B respectively illustrate a line segment before andafter deletion of 4-connected pixels.

FIGS. 14A-14C illustrate the deletion of junction pixels.

FIGS. 15A and 15B respectively illustrate a segment before and after thejunction has been removed.

FIG. 16 illustrates the overall operation of the busy mask processingblock of FIG. 6.

FIG. 17 illustrates the processing performed by the line extractionprocessing block of FIG. 6.

FIGS. 18A-18D illustrate the edge thinning templates employed inaccordance with exemplary embodiments of the present invention.

FIGS. 19A and 19B respectively illustrate a plurality of pixels beforeand after edge thinning.

FIGS. 20A and 20B respectively illustrate a thinned edge representationwith no noise and a thinned edged representation with noise.

FIG. 20C illustrates the segmentation of a thinned edge with noise inaccordance with exemplary embodiments of the present invention.

FIGS. 21A and 21B respectively illustrate a region with a slightcurvature and a best fit line for the region with a slight curvature.

FIG. 22 illustrates the processing performed by the segmentationprocessing block of FIG. 6.

FIG. 23 illustrates the templates for vertical and horizontal pointdeletion or gap filling.

FIG. 24 illustrates exemplary processing performed by cloud bankidentification processing block of FIG. 6.

FIG. 25 illustrates a functional flow of the cloud refinement processingblock of FIG. 6.

FIG. 26 illustrates the large scale processing performed to identifybodies of water in accordance with exemplary embodiments of the presentinvention.

FIG. 27 illustrates the processing performed in parallel edge processingblock of FIG. 26.

FIG. 28 illustrates the processing performed on the medium resolutionimage for identification of bodies of water in accordance with exemplaryembodiments of the present invention.

FIGS. 29A and 29B respectively illustrate the filter employed in thevariance operator processing block of FIG. 28, and the calculation ofthe values of the filter.

FIG. 30 illustrates the various processing performed on the small scaleimage.

FIG. 31 illustrates the processing performed by terrain identificationprocessing block of FIG. 30.

FIG. 32 illustrates an exemplary morphological filter in accordance withexemplary embodiments of the present invention.

FIGS. 33A and 33B respectively illustrate the components employed in thedilation processing and the components employed in the erosionprocessing of the morphological filter in accordance with exemplaryembodiments of the present invention.

FIGS. 34A and 34B respectively illustrate the Kirsch edge operatortemplates and the determination of the sign of the maximum magnitude.

FIG. 35 illustrates a method for determining whether line segments arecollinear.

FIG. 36 illustrates the processing performed in the determination ofwhether two lines are parallel in accordance with exemplary embodimentsof the present invention.

FIG. 37 illustrates the grouping of parallel line pairs in accordancewith exemplary embodiments of the present invention.

FIG. 38 illustrates the processing performed by the identification andnomination processing block of FIG. 30.

FIG. 39 illustrates the extension of perpendicular line segments.

FIG. 40 illustrates the extension and reflection of two perpendicularline segments.

FIG. 41A illustrates extracted lines from an image from a top view of asimple perfect building.

FIG. 41B illustrates an exemplary illustration of lines actuallyextracted from an image.

FIG. 41C illustrates the reflection and extension of line segments ofthe image extracted in FIG. 41B.

FIG. 42A-42C illustrate a structure confidence feature which reduces theconnection of a building wall to a shadow line in accordance withexemplary embodiments of the present invention.

FIGS. 43A and 43B respectively illustrate rectangular objects whichoverlap with two other rectangular objects and a representation of theserectangular objects employing only the outside edges of thenon-overlapping regions of the rectangular objects.

FIG. 44 illustrates the nomination of objects using a Bayes classifier.

FIG. 45 illustrates a generic process employed in the identification oflines of communication in small and mid scale resolution imagery.

FIG. 46 illustrates the processing performed for the identification oflines of communication in the small scale imagery.

FIGS. 47A and 47B illustrate mappings between image space and Houghspace.

FIG. 48 illustrates the mapping function for mapping from image spaceinto Hough space.

FIG. 49 illustrates clutter rejection processing in accordance withexemplary embodiments of the present invention.

FIG. 50 illustrates the processing performed to detect lines ofcommunication in mid scale resolution imagery.

FIG. 51 illustrates the tuned filter employed in accordance withexemplary embodiments of the present invention.

FIG. 52 illustrates the cleaning filter employed to remove small regionsin accordance with exemplary embodiments in the present the presentinvention.

FIG. 53 illustrates thinning templates in accordance with exemplaryembodiments of the present invention.

FIG. 54 illustrates a bounding box in accordance with exemplaryembodiments of the present invention.

FIG. 55 illustrates the application of the Hough transform to an imagein accordance with exemplary embodiments of the present invention.

FIG. 56 illustrates a high level block diagram of the contextual linereasoning processing in accordance with exemplary embodiments of thepresent invention.

FIG. 57 illustrates the processing performed in the contextual linereasoning processing.

FIG. 58 illustrates the processing steps for connecting primitive linesegments.

FIGS. 59A-59C illustrate various relations between line pairs.

FIG. 60 illustrates density clutter rejection processing in accordancewith the present invention.

FIG. 61 illustrates the relation between various distance matrices inaccordance with exemplary embodiments of the present invention.

FIGS. 62A-62E illustrate a best path analysis in accordance withexemplary embodiments of the present invention.

FIG. 63 illustrates zigzag clutter rejection processing in accordancewith exemplary embodiments of the present invention.

FIG. 64 illustrates vector group connection processing in accordancewith exemplary embodiments of the present invention.

FIG. 65 illustrates the connection of vector groups in accordance withexemplary embodiments of the present invention.

FIG. 66 illustrates processing performed foreseen context analysis inaccordance with exemplary embodiments of the present invention.

DETAILED DESCRIPTION Overview

FIG. 1 is a block diagram of a system 100 for processing image data inaccordance with exemplary embodiments of the present invention. Thesystem 100 includes a detector 105, a processor 120, a memory 180, adisplay 190, a storage unit 192 and a communication interface 194. Theprocessor includes an image processing block 125 and an algorithmselector 130. The display 190 can be any type of display for outputtinginformation in a human readable form including a cathode ray tubemonitor, an LCD monitor, a printed piece of paper, or the like. Further,processor 120 can be hard-wired circuits, or a processor executing asuitable set of program instructions stored on a computer readablestorage medium such as a random access memory (RAM), read only memory(ROM), magnetic storage medium (such as magnetic tape, disk or diskette)or optical storage medium (such as compact disk (CD) ROM).

Detector 105 can comprise an infrared imaging device, a thermal imagingdevice, a regular photographic device or the like. Memory 180 can be anytype of memory including random access memory (RAM) electronicallyerasable memory (EPROM), or the like. Storage unit 192 can be any typeof storage including magnetic or optical drives, a local drive or anetwork drive, and a floppy disk, hard drive, CD-ROM, DVD-ROM, DVD-RAM,or a tape drive. Communication interface 194 can comprise any type ofinterface for connecting to a communication network, such as a data orvoice network, a land-line or wireless network, or the like. It will berecognized that one of ordinary skill in the art would understand how tobuild a communication interface, and hence, a detailed description ofthis interface is omitted.

Detector 105 captures an image of a scene and provides the image toprocessor 120. In addition to the captured scene, the detector canprovide metric data associated with the captured scene. The metric datacan include geographic location, terrain type, ground sample distance,weather, viewing conditions, band frequency of the sensor band, degreesof freedom of the sensor, viewing angles, and/or positional vector. Theimage processing block 125 receives the captured scene and processes theassociated image data using one or more processing algorithms to produceone or more processed signals. Specifically, as will be described inmore detail below, the processing algorithms can be specificallydesigned to identify various objects in the image data such as bodies ofwater, vehicles, buildings or communication lines.

The processed signals are input to the algorithm selector 130. Thealgorithm selector automatically selects among the processed signals,each of the processed signals being associated with a differentdetection algorithm, based upon predetermined conditions associated withthe captured scene, e.g., weather, time of day, type of terrain andtemperature. The processor then outputs the selected processed signalsto display 190, storage unit 192 and/or communication interface 194. Formore information regarding the operation of the algorithm selector theinterested reader should refer to U.S. patent application Ser. No.10/196,168 filed Jul. 17, 2002 “Algorithm Selector”, the entire contentsof which is herein expressly incorporated by reference.

FIG. 2 is a block diagram of the various functional processing blocks ofimage processing block 125. As illustrated in FIG. 2, the presentinvention operates using three different resolutions of the receivedimage. Specifically, small scale processing is performed on the image inits original resolution or first resolution, mid scale processing isperformed on a downsampled version of the original image at a secondresolution, and large scale processing is performed on a downsampledversion of the image at the second resolution in a third resolution. Inaccordance with exemplary embodiments of the present invention, largescale processing is performed first, mid scale processing is performednext, and small processing (except for processing block 215) isperformed last. By performing the processing in this order, informationobtained in the higher scale processing (e.g., a cloud mask) can beemployed in lower scale processing (e.g., small object detection).

As illustrated in FIG. 2, the small scale processing begins with theinput of an image with a first resolution 205, and support data andcartographic information 210. This information is input into a terraintype identification processing block 215 which identifies terrain typespresent in the image. In accordance with exemplary embodiments of thepresent invention, the type of terrain is identified by employing graylevel co-occurrence. Continuing with the small scale processing, theoutput of the terrain type identification processing is input to a smallobject detection block 220. Next, linear objects are detected byprocessing block 225. The linear object detection processing block 225provides an output identifying roads, trails, paths, bridges, airportsand the like. The small scale lines of communication processing blockoutputs an identification of roads, power lines, communication lines,gas lines, oil lines, and the like. The building and structure detectionblock 235 outputs an identification of buildings, structures, compounds,agriculture and the like.

The mid scale processing is performed on a downsampled version of theimage with a first resolution 205 using an output of four-to-onedownsampling block 240. The output of four-to-one downsampling block 240is an image at a second resolution which is received by a mid scaleriver mask processing block 245. Mid scale river mask processing block245 outputs an identification of river regions. The mid scale lines ofcommunication processing block 250 outputs an identification ofhighways, roads, trails and the like.

The large scale processing is performed on a downsampled version of themid scale imagery. This is achieved by downsampling the image at asecond resolution using the four-to-one downsampling processing block255 to produce an image at a third resolution. The downsampled imageryis then provided to cloud mask processing block 260. Cloud maskprocessing block 260 outputs an identification of cloud regions. A largescale river mask processing block 265 outputs an identification of riverregions. The activity detection processing block 270 outputs anidentification of roads, trails, buildings, structures, agriculture andother manmade activity.

The output of processing the imagery at the various resolution levels iscombined in a contextual line reasoning processing block 275 and in ascene contextual analysis processing block 280. The contextual linereasoning processing block 275 and the scene contextual analysisprocessing block 280 receive images identifying objects in the imagewith a first resolution from the various processing blocks in FIG. 2 andemploy a variety of criteria to improve the confidence in the detectedobjects. It should be recognized that FIG. 2 is merely intended as anoverview of the various processing performed by image processing block125. Accordingly, the order in which the various processing blocks inFIG. 2 are illustrated within the particular scale of processing is notnecessarily the order in which these processes are performed. Theparticular order of the processing will be evident from the relationshipof the various processing blocks in the description below. Now that anoverview of the entire system has been presented, a detailed descriptionof the various processing blocks illustrated in FIG. 2 will now bepresented.

Gray Level Cooccurrence

As illustrated in FIG. 2, the present invention employs a gray levelcoocurrence processing to identify terrain types present in the originalimage. Gray level cooccurrence is an image processing operator used toderive a set of features that can determine texture properties of animage. The gray level cooccurrence is defined as a matrix P(i,j,d,θ)made up of probabilities that gray level i occurs near gray level j, ata distance d pixels apart, with an angular orientation of θ degrees. Inpractice, multiple θ values are used, and one gray level cooccurrencematrix is computed for each of these θ. Multiple d values can also beused, in which case, a set of matrices is computed for each d value.FIG. 3 illustrates four θ neighborhoods which are applied over an inputimage. In FIG. 3, C is the pixel that determines the i index value ofthe matrix, and X is the pixel that determines the j index value of thematrix. Although there are four additional θ values that are reflectionsof the four neighborhoods illustrated in FIG. 3, there is no need tocompute separate matrices for these since they can be combined with thecomputation of their symmetric counterpart by incrementing the (j,i) binof the gray level cooccurrence matrix at the same time as the (i,j) isincremented in the matrix. Dividing each individual matrix cell by thesum of all cell values normalizes the gray level cooccurrence matrix.The value in each (i,j) cell then becomes the probability that the (i,j)gray level pair occurs in the image.

The gray level cooccurrence is employed in the present invention in thedetections of clouds in the large scale imagery and in general terraintype detection for small scale imagery. Gray level cooccurrence uses thespatial relationship of individual pixel gray values in the image toproduce a set of texture feature measurements for a particular imageregion. The present invention employs these feature measurements inmulti-class statistical classification to determine the terrain class ofunknown regions in an image.

FIG. 4 illustrates the overall processing flow employed in the presentinvention. Since the pixel values in the image can be as high as 65,535,or 16 bit values, the gray level cooccurrence matrix would be too largeto fit into a memory. To remove this problem the pixels are rescaled to8 bits, or 255 valued numbers in processing step 410. This results in amore manageable gray level cooccurrence matrix. The rescaling can beaccomplished with the following formula:

${I_{out}\left( {i,j} \right)} = {\frac{{I_{i\; n}\left( {i,j} \right)} - \min}{\max - \min} \times 255}$

After resealing the image is split into fixed size windows that are128×128 pixels in processing step 415. Each window is then individuallyprocessed in accordance with the remainder of the processing stepsillustrated in FIG. 4. A set of gray level cooccurrence matrices istallied for each 128×128 window, these matrices including the fourmatrices for the four angles illustrated in FIG. 3. Next the matricesare processed by the functions illustrated in FIG. 5. These functionsinclude energy, entropy, contrast, inverse difference moment, andcorrelation. This produces five sets of four values each. These fivesets of values are next converted into gray level cooccurrence featuresin processing step 430. Specifically, two features are calculated forfive sets, namely, the mean, and range (maximum value minus minimumvalue) of each of the four sets. This results in a total of ten graylevel cooccurrence features for each window. Finally, each window isclassified into the various terrain types employing the gray levelcooccurrence features using a statistical classifier. One skilled in theart will recognize how to make and use such a statistical classifier.

Clouds and Manmade Activity Detection

The processing performed for the cloud mask processing block 260 and theactivity detection processing block 270 are similar, and hence, will bepresented together. Processing which differs between these processingblocks will be highlighted in the description below. FIG. 6 is anoverview of the various processing employed by cloud mask processingblock 260 and activity detection processing block 270. The image with afirst resolution and metric data 605 are received by these processingblocks and the image is downsampled in processing block 610. Thedownsampling produces an image at a third resolution 615 which isprovided to a primitive extraction processing block 620. Since the cloudand activity detection is performed on large scale images, the imagewith the first resolution is downsampled twice by processing block 610.The primitive extraction processing block 620 processes the image datato produce a second image which identifies areas in the image whichborder regions of different intensities. Next, a busy mask processingblock 625 processes the downsampled image to produce a third image whichidentifies portions of the input image for which an average gradientmagnitude of the portion is greater than a threshold level.

Next, line extraction processing block 630 processes the second imagewhich is output by primitive extraction block 620 to produce a fourthimage which identifies lines in the image. The third image, output fromthe busy mask, is then updated with the fourth image to preserve thelinear features. Hence no linear features are included in the busy mask.This image is then segmented into a plurality of regions by segmentationprocessing block 635. Next, the background regions are determined andmerged in processing block 640. The objects that are desired to bedetected are identified by feature extraction processing block 645. Theextracted features are then classified in processing block 650. It is atthis point that the activity detection processing block would output anidentification of roads, buildings, agriculture and other types ofmanmade activity. However, the cloud mask processing block would performfurther processing using the output of classification processing block650. Specifically, the output of classification processing block 650 isemployed to identify cloud banks in processing block 655, and thedetection of clouds is refined in processing block 660.

FIG. 7A illustrates an exemplary technique for downsampling the imagewith a first resolution. Specifically, the image with a first resolutionis downsampled by employing a 6×6 low pass convolution filter. Thisfilter is effective in removing aliasing caused by high frequencies. Thefull 6×6 convolution filter is illustrated in FIG. 7B, which is acombination of the X direction filter of FIG. 7C and the Y directionfilter of FIG. 7D. In order to reduce processing operations, the presentinvention decomposes the full 6×6 convolution filter kernel into an Xdirection filter and a Y direction filter. Accordingly, the originalresolution image is initially low pass filtered in the X direction 750.Next, the original resolution image is downsampled in the X direction755 and then low pass filtered in the Y direction 760. Finally, theoutput of the low pass filtering operation in the Y direction isdownsampled in the Y direction 765. The output of this downsamplingprocess can be input into an identical downsampling processing block iflower resolution imagery is desired. Since the identification of cloudsand manmade activity detection is performed using large scaleprocessing, the original resolution image at a first resolution isdownsampled twice.

FIG. 8 illustrates an overview of the processing performed by theprimitive extraction processing block 620. The primitive extractionprocessing block initially extracts edges from the image at a thirdresolution by processing block 805. Processing block 810 performs athinning operation on the extracted edges. Next, the gradient magnitudesof the pixels are maximized in processing block 815. The maximizedgradient magnitudes are then compared to a local mean threshold inprocessing block 820. Processing block 820 outputs an image of theidentified primitives.

The edge extraction processing block 805 is employed to identify areasof the image which border regions of different intensities using a Sobeloperator. The areas of an image which are identified by a Sobel operatoras bordering regions of different intensities are referred to in the artas edges. Accordingly, it is common in the art to refer to the operationperformed by a Sobel operator as edge extraction. The Sobel operator isa conventional image processing routine which is an approximation to thegradient (edge magnitude) of a digital image and can be computed using atemplate operator. FIGS. 9A and 9B illustrate the Sobel templates.Specifically, the vertical template is convolved with the image toproduce a horizontal gradient D_(x). The image is also convolved withthe horizontal template to produce a vertical gradient D_(y). Thegradient magnitude for each pixel is calculated by summing a square ofthe horizontal gradient with a square of the vertical gradient andtaking the square root of the sum. An arc tangent operation of theresult of a division of the vertical gradient by the horizontal gradientprovides the gradient direction. In accordance with exemplaryembodiments of the present invention, the Sobel direction is mapped toeight directions in accordance with the directional assignmentsillustrated in FIG. 9B. It will be recognized that pixel locations wherethe gradient magnitude is high represents transition areas or edgelocations in the image. The result of the processing using the Sobeloperator is the generation of an image which identifies edges, i.e.,areas of the image which border regions of different intensities. Oncethe edges are extracted, the mean, maximum, minimum and standarddeviation of the gradient magnitude image is calculated. Thisinformation is used in the busy mask processing block 625 and in thesegmentation processing block 635 as will be described in more detailbelow.

The edge thinning processing block 810 performs an edge thinningoperation on every pixel to preserve maximal edge responses.Effectively, the non-maximal responses in a direction normal to the edgeresponse, i.e., across the edge, are suppressed. FIG. 10 illustrates thefour templates employed for this edge thinning processing. Specifically,one of the four templates illustrated in FIG. 10 is applied at eachlocation in the gradient magnitude image. The gradient direction, onethrough eight, at any given pixel location determines which template touse. If the gradient magnitude of the center pixel is less than thegradient magnitude at any of the locations in the template marked by an“X”, then the center pixel is a non-maximal edge and it is not includedin the edge map image, i.e., it is set to a value of zero. If, however,the center pixel's gradient magnitude is greater than, or equal to, theedge magnitude at the “X” positions of the template, then the centerpixel represents a locally maximal edge response and its magnitude isincluded in the edge map image.

The edge map image output from edge thinning processing block 810 isprovided to gradient magnitude maximization processing block 815. Inthis processing block, the gradient magnitudes of the pixels aremaximized so that each pixel on a line has its gradient magnitude set tothe value of the pixel with the greatest gradient magnitude on the sameline. These new gradient magnitudes are then fed into a thresholdroutine. The threshold routine first performs a line segment cleaningprocedure to thin the line segments and to delete the four connectedpixels. Next, the junction pixels and their immediate neighbors aretemporarily removed. Then all pixels on the same line segment areassigned the maximum gradient magnitude value of that line. The finalstep of this process restores the deleted junction pixels and theirneighbors. These steps are now described in detail.

The line segment cleaning procedure converts the edges in the edge mapimage into a set of connected pixels that form line segments, althoughnot necessarily straight line segments. The line segments must have aproperty that each has only two endpoints, unless it is a closed curve,in which case it has no endpoints. To achieve this, the edge pixels arethinned, four-connected pixels are deleted, and junction pixels aredeleted.

The edge thinning processing often leaves edges that are more than onepixel wide. To reduce the width of these edges to a single pixel, athinning process is employed that repeatedly “peels” pixels for eachside of the edges until a one pixel wide line remains. This thinningprocess is illustrated in FIGS. 11A and 11B. Specifically, FIG. 11Aillustrates a line segment before edge thinning and FIG. 11B illustratesa line segment after edge thinning.

As illustrated in FIG. 11B, the thinned edges are both four-connectedand eight-connected. To facilitate the identification of junctionpixels, the four connected pixels are removed if this does not cause abreak in the edge connectivity. FIG. 12 illustrates the patternsemployed to remove the four-connected pixels. In the patterns of FIG.12, a non-zero pixel, i.e., a “1” location, in the image is deleted,i.e., set to zero, if it fits one of the four patterns. FIG. 13Aillustrates a line segment before deletion of four-connected pixels andFIG. 13B illustrates a line segment after the deletion of four-connectedpixels.

FIGS. 14A-14C illustrate the deletion of junction pixels. The numbers inFIGS. 14A-14C represent the gradient magnitudes of particular pixels inan image. As illustrated in FIG. 14A, three-connected groups of edgepixels frequently come together at a point. One of the three groups isdeleted so that only line segments that can be identified by a pair ofend points remain. Since most of the junction points are due to noise,the edge group that has the smallest gradient magnitude at the junctionis disconnected, as illustrated in FIG. 14B. The four-connected pixelremoval described above is then performed which results in the pixelarrangement illustrated in FIG. 14C.

After line segment cleaning is performed in the magnitude maximizationprocess, the junction pixels and their immediate neighbors aretemporarily removed. FIGS. 15A and 15B illustrate a segment before andafter the junction has been removed. In FIGS. 15A and 15B “J” representsa junction pixel, “N” represent a pixel within a 3×3 neighborhood of thejunction pixel, and “X” represents non-background pixels. To determinewhether a pixel is a junction pixel, its eight neighbors are checked ina circular fashion. If more than two transitions from background to edgeto background are found, the center pixel is determined to be a junctionpixel. It will be recognized that a background pixel is one whosegradient magnitude is zero. Using the junction illustrated in FIG. 15A,first the junction pixel and all of its neighbors in its 3×3neighborhood are removed producing the result illustrated in FIG. 15B.The information regarding the junction pixel and the pixels within its3×3 neighborhood is saved so that these pixels can be restored later inthe processing.

Next, all pixels on the same line segment are set to the highestgradient magnitude value of that line segment. This is performed bylabeling the line segments, finding the maximum gradient value for eachlabeled line, and then setting all pixels with the same label to thatmaximum gradient magnitude value.

As discussed above, the information regarding the deleted junctionpixels and their 3×3 neighbors was saved. Accordingly, the deletedjunction pixels and their neighbors are restored, at the same timesetting their gradient magnitude to the highest value in each pixel's3×3 neighborhood. This step ensures that the previously deleted pixelshave the maximum value of the line segment to which they are connected.

The last step of the primitive extraction processing includes applying alocal mean gradient magnitude threshold to delete weak edges in theimage. This threshold is a percentage of the mean gradient magnitudevalue for an N×N neighborhood. The size of the neighborhood depends onthe resolution of the image. The percentage threshold is related to thesize of the neighborhood. For example, if the resolution is greater than1.85 meters, a 16×16 neighborhood and a threshold percentage of 3.5 areused. Otherwise, a 32×32 neighborhood is employed with a thresholdpercentage value of 3.0. Accordingly, the image output from magnitudemaximization processing block 815 is processed such that the gradientmagnitudes of each pixel in the N×N neighborhood are set to the meangradient magnitude for the N×N neighborhood, thereby generating a localmean image. The percentage value is then multiplied with the local meanimage to obtain a scaled local mean image which is used to threshold theimage output from magnitude maximization processing block 815. Thosepixels that have a corresponding gradient magnitude below the thresholdvalue are deleted in the image output from magnitude maximizationprocessing block 815. This ends the processing for the primitiveextraction processing block 620.

The majority of the low level processing algorithms have a goal ofextracting the most important edges or lines while removing much of thenoise or clutter pixels. This is usually achieved using a techniqueknown as thresholding. However, it has been recognized that to achieveoptimal results a single image may require different thresholds fordifferent areas or sub-images. An algorithm processing an image shouldbe aware of these requirements in order to perform the required dynamicthresholding. The present invention employs a busy mask to denote areasthat require higher gradient magnitude thresholds than the blander areasof the image. In other words, the busy mask processing block 625generates a third image identifying portions of the image for which anaverage gradient magnitude of the portion is greater than a threshold.The end result is a more accurate output image with less noise andclutter pixels.

FIG. 16 illustrates the overall operation of the busy mask processingblock 625. The present invention employs a gradient operator to measurethe “busyness” for each pixel compared to its neighbors by computing thegradient magnitude of each pixel. The greater the gradient magnitude,the busier the texture. Accordingly, one input to the busy maskprocessing block is the gradient magnitude image created from the Sobeloperator. A second input is the statistics file generated by theprimitive extraction processing block 620. As illustrated in FIG. 16,the statistics file and gradient magnitude image are input to processingblock 1610 where an average of the gradient magnitudes is computed overan N by N neighborhood to obtain a local measurement of the gradientmagnitudes for a particular portion of the image. The window size forthe local mean processing is determined based on the resolution of theimagery, and can be either, for example, a 16×16 or a 32×32. The outputof the local mean processing block 1610 is input to threshold processingblock 1620. The threshold processing block 1620 compares the values ofthe local mean for each portion of the image to a global mean of each ofthe local means. The output of the threshold processing block 1620 isinput to processing block 1630 where any small holes in the mask arefilled by a series of 3×3 dilations, i.e., expansions. The output of theexpand and fill processing block 1630 is input to size evaluation block1640. The size evaluation block 1640 removes portions of the image thatare less than a predetermined size, e.g., less than 5,000 pixels. Theresult of the size evaluation processing block 1640 is output as a busymask, i.e., a third image identifying portions of the image for which anaverage gradient magnitude of the portion is greater than a threshold.

After creating the busy mask, the line extraction is performed togenerate a line mask image. Specifically, having completed the gradientmagnitude thresholding in the primitive extraction processing block 620,the stronger edges are then converted into straight lines. FIG. 17illustrates the processing performed by line extraction processing block630. Initially, sensor artifacts are eliminated in processing block1705. These sensor artifacts are removed since they can significantlyskew the edge statistics and line analysis. The sensor artifacts aredetected and eliminated based upon the type of sensor being processed.For example, for scanned sensor arrays artifacts appears as long linesegments in the scan direction. For steering arrays, scan sensorartifacts appear as fixed pattern noise. Accordingly, in either case theartifacts are detected and eliminated from the extracted edge imageusing known techniques.

To reduce the edge data and to create a binary mask of the strongestedges in the original resolution image, processing block 1710 appliesgradient magnitude threshold to the gradient vector image received fromprocessing block 1705. Processing block 1710 also receives a terrainmask image produced by terrain identification processing block 215.Specifically, processing block 1710 employs the terrain mask image todetermine a local threshold for each particular type of terrain in theimage. The threshold is a percentage of the mean gradient magnitudes foreach terrain region, and can be for example, a value of 1.5%. Thosepixels which have a corresponding gradient magnitude below the thresholdare deleted. The terrain mask image provided by terrain identificationprocessing block 215, in addition to an identification of terrain type,can contain information on clouds, rivers, river edges, and vegetationin the image. This information is employed to mask the cloud, river,river edges, and vegetation in the image to prevent false nominations ofman-made activity. However, since information about clouds and riversmay not have been determined yet, the terrain mask may comprise only theidentification of the terrain types present in the image.

As discussed above, the present invention employs a local threshold toprevent the deletion of objects, and to reduce clutter. For example, ifa region in an image were composed of forest or rocks with an object inan adjacent field, and a global gradient magnitude threshold were used,the threshold might remove the object because the busyness of the forestor rocks would make the threshold extremely large. Similarly, byemploying a global gradient magnitude threshold, if a region is blandwith the exception of a few rocks, the threshold will be low which willallow more clutter through in the rocky area.

Processing block 1715 receives the thresholded gradient vector image tothin the remaining edges in the image. At each edge pixel one of thefour templates illustrated in FIGS. 18A through 18D is employed,depending upon the gradient direction of the particular pixel. Asillustrated in FIGS. 18A through 18D, each template is employed for twoedge directions. In FIGS. 18A-18D “e” represents an edge pixel and “x”represents pixels which are examined for the thinning procedure.

FIGS. 19A and 19B respectively illustrate a plurality of pixels beforeand after edge thinning. In FIG. 19A each remaining gradient vectorpixel has a gradient direction of 7. Accordingly, at each gradientvector pixel the template illustrated in FIG. 18A is employed.Specifically, at each edge pixel the pixels to the left and right of theparticular pixel are examined. If the pixel to the left or to the rightdoes not have a gradient direction of 7, this pixel is set to zero andnot further considered. The pixels to the left or right which have thesame gradient direction as the pixel being considered are examined todetermine their gradient magnitude. If the gradient magnitude of thepixel being considered is greater than or equal to both the pixel on theleft and the right which has the same gradient direction as the pixelbeing considered, the pixel being considered is retained. If the pixelbeing considered does not have a gradient magnitude greater than orequal to both the left and right pixels with the same gradientdirection, then the gradient magnitude for that pixel is eliminated fromthe output image.

As illustrated in FIG. 19A, in the top row the gradient magnitude of 6is greater than the gradient magnitude of 2; in the next row thegradient magnitude of 24 is less than the gradient magnitude of 32,which is greater than the gradient magnitude of 5; the gradientmagnitude of 34 is greater than the gradient magnitudes of 2 and 5; andthe gradient magnitude of 25 is greater than the gradient magnitude of3. Accordingly, in the top row of the gradient magnitude of 6 isretained, in the next row the gradient magnitude 32 is retained, in thenext row the gradient magnitude 34 is retained, and in the last rowillustrated in FIG. 19B the gradient magnitude 25 is retained.

After the thinning operation has been performed on every pixel in thegradient vector image, an image with the remaining thinned edges ispassed to processing block 1720. Processing block 1720 attempts toconvert the thinned edges into straight line segments by linking thethinned edges. The present invention employs the gradient directions inthe attempt to link the edges. Ideally, a straight horizontal edge willconsist only of gradient directions labeled “1”, as illustrated in FIG.20A. However, noise may be present in the image. Accordingly, asillustrated in FIG. 20B, a horizontal edge consisting of only directionslabeled “1” may appear to have other directions present, e.g., “2” or“8”. To prevent the identification of many small regions in the image,in view of the presence of noise in the image, the present inventiongroups like directions into similar regions. Accordingly, a gradientdirection labeled with direction “1” can be grouped with either agradient direction of “2” or “8”. If this grouping were not performed,then the noise would result in many small regions rather than a fewlarge regions. For example, as illustrated in FIG. 20C, due to the noisethe edge directions illustrated in FIG. 20B would be broken down intoseven different regions. To overcome this problem, as illustrated inFIG. 20B, two non-overlapping regions are created. Non-overlappingregion “A” is formed from all gradient directions of “1”, “3”, “5”, and“7”. Similarly, non-overlapping region “B” comprises gradient directions“numeral 2”, “4”, “6”, and “8”. Region “A” will be selected since itcontains twelve “1”s, while region “B” contains 10 “8”s. The forming ofregions performs a smoothing process that assists in removing some ofthe noise in the linked edges.

Processing block 1725 receives the gradient vector image with the linkededges from processing block 1720 to breakup any curved lines. It hasbeen recognized that regions with a slight curvature may be incorrectlyrepresented in the image as a straight line. Specifically, when groupinggradient directions, edges with gradual curves may not be properlyseparated resulting in one long region, rather than several shortregions, being extracted in the image. To prevent this regions arebroken into 30 pixel regions and then the best-fit line is computed forthe small regions. If an edge is straight and longer than 30 pixels, theedge will be detached into several lines no longer than 30 pixels inlength, but will be reconnected in subsequent processing steps asdescribed below.

Once the edges are broken into 30 pixel segments a straight lineapproximation can now be calculated for each region. This straight lineapproximation is computed by calculating the slope and center of massfor each region. A straight line that best fits the calculated slope andcenter of mass is created. This reduces each region into a straight lineof a single pixel width. FIG. 21A illustrates a region with a slightcurvature, while FIG. 21B illustrates the straight line approximation ofthis region. Once the line segments are extracted, pertinent data foreach line is calculated, including the slope, the end points, theY-intercept, and length. This data is then placed into a line segmenttable. For optimal processing, line segments are organized by theirslope as a value between 0 and 179 degrees. Line strength is a featurethat measures the percentage of pixels on a line segment which passesthe gradient magnitude threshold. The line strength feature is helpfulin deleting clutter lines. A metric is incorporated that takes care ofline length, allowing for greater slope tolerance in shorter linesegments. An image containing the shorter cleaner line segment is outputfrom processing block 1730, thereby ending the processing of lineextraction processing block 630.

After line extraction processing has been performed, segmentationprocessing is performed. The present invention employs an edge-basedsegmentation technique and the Perkins segmenter to performsegmentation. Two key parameters with which drive the performance arethe edge threshold and the minimum region size. Typically, a programmermust commit to some static configuration of these parameters prior toprocessing the image. This will lead to non-optimal results since eachimage or image sub-area will have its own optimal segmenter arrangement.Accordingly, by employing an adaptive segmentation technique thatautomatically adjusts the parameters to suit the particular scene in theimage, the present invention will provide superior performance to thatof a non-adaptive version of the same technique.

FIG. 22 illustrates the processing performed by segmentation processingblock 635. In general, the segmentation processing block segments theimage into a plurality of regions. Each region will either be an objectof interest or a background region. As illustrated in FIG. 22, theinputs to the segmenter are the gradient magnitude image, the gradientdirection image, the busy mask image, and the line mask image. The firststep in the segmentation processing is to threshold the gradientmagnitude image in order to delete the noise and clutter pixels.Processing block 2210 employs a two step threshold approach. First, alocal mean operator is used to delete the weak edges in a 5×9neighborhood. The line mask image is employed to preserve linearfeatures that may be on weak edge segments. For each neighborhood in thegradient magnitude image, the local mean is calculated and all pixelswith a gradient magnitude less than a certain percentage of the localmean, and which are not masked by the line mask image, are deleted. Thecorresponding pixels in the gradient direction image are removed aswell. Next, a global threshold is applied to the gradient magnitudeimage. The present invention employs two distinct global thresholds, onefor pixels in the busy mask area and one for bland areas of the image.The thresholds are a percentage of the mean gradient magnitude value forthe entire image. The segmenter automatically determines the percentageused. The line mask image is again used to preserve weak lines that arepossible linear features. Those pixels that have a correspondinggradient magnitude below the threshold value and do not have acorresponding pixel in the line mask image are deleted. The associatedpixels in the gradient direction image are also removed.

Next, vertical and horizontal gaps of one pixel between edges areselectively filled by processing block 2220. The templates illustratedin FIG. 23 are employed to fill the gaps. Specifically, if any of thetemplates match the image data, the center pixel value is set to a “1”.Once these gaps have been filled, isolated edges less than 4 pixels inlength are deleted.

Processing block 2230 fills small gaps and preserves small regions ofuniform intensity. The present invention employs an iterative cycle ofexpansion, labeling, and contraction of the edge pixels. A region is agroup of one or more non-zero adjacent pixels. In accordance with thepresent invention, active edge points are edge pixels that touch onlyone region, i.e., edge pixels which are isolated. Inactive edge pixelsare adjacent to two different regions. Initially, all edge points areactive. At the beginning of each iteration, the edge pixels that areconnected to two different regions are changed to inactive.

Expansion involves dilation of the active edge points to close smallgaps. In this step, pixels being changed to edge points are labeled astemporary edge points. The resulting connected regions are then given aunique label and finally the temporary edge points are contracted, i.e.,thinned. It is important that the edges are thinned to a single pixelwidth so that when an edge point is along the boundary of two regions itwill be immediately adjacent to pixels of both regions. The temporaryedge points can be removed in the contraction operations if they havenot been successful in closing a gap in the boundary. Next, the regionsare uniquely labeled, e.g., numbered, and small regions less than anautomatically determined number of pixels are eliminated in processingblock 2240.

The last step in the segmentation process involves the determination ofwhether the gradient magnitude image is properly segmented. This isperformed to prevent over-segmentation from occurring. Accordingly,after segmenting the image, processing block 2250 determines the numberof regions obtained. If this is greater than a reasonable number, thensmaller regions, which usually represent edge noise, are merged into thebackground in processing block 2260. A value which represents areasonable number of regions is dependent upon the number of pixelswhich make up the image and upon how busy the image is. In an exemplaryembodiment, the reasonable number value can be set using the range250-500. Next, a new region count is determined. This check ensures thatweaker edges that are part of larger regions are preserved withoutover-segmenting the image. The output from the segmentation processingblock is an image of labeled regions, sequentially assigned values fromone to the total number of regions segmented.

As described above, the edge thresholds and the minimum region size areautomatically determined parameters for the Perkins segmentationprocess. A rule base is employed in the segmentation process to derivethese values. The edge thresholds are dependent upon the averagegradient magnitude strength of the entire image at the originalresolution, the busy mask, and the terrain type. The edge threshold is apercentage of the average gradient magnitude strength. To determine theedge threshold, the higher the average gradient magnitude strength, thelower the percentage, and hence, a lower edge threshold used. Theminimum region size is based upon the resolution of the image and canrange from 5 to 50 pixels.

After segmenting the image, background regions are determined andvarious regions are merged by processing block 640. The segmentationprocess results in the entire image being divided into separate regions.The next step is to determine which of those regions are actuallybackground regions, thereby leaving only those non-background regions tobe run through the rest of the processing. Normally, the backgroundregion will be the largest region in terms of number of pixels. However,this may not always be the case. Accordingly, in the present invention,a check for “spanning” regions that are segmented due to edges that passfrom one image border to another is performed. Hence, the backgroundregions are determined based on their area and their adjacency to theimage borders. Regions that span from top to bottom or from left toright are assigned as background. A region that is located at a corneris assigned as background if it is larger than a percentage of theimage, the percentage based upon the image resolution. The result ofthis process ensures all background regions are labeled the same uniquenumber, i.e., each pixel in a background region is set to the value “1”.

Many edge-based segmenters tend to break larger regions into severalsmall regions, especially when the thresholds are set low to extractaccurate boundaries for all regions of interest. Therefore, aftersegmentation and background determination, the present invention mergesadjacent regions into one unique region. This technique consists ofmarking the outer edges of adjacent regions, and then removing the inneredges between them to form the new merged region. Region merging isperformed to avoid disrupting the desired boundaries, while reducing thenumber of regions to process. Accurate boundaries result in featurecomputations that are more exact, and therefore, allow for more correctclassifications. Another reason the region merging technique is employedin the present invention is to obtain a more precise area calculationfor the object of interest.

After region merging, the objects of interest will be located in regionswhich have been labeled as being unique from the background. In additionto the objects of interest, clutter objects may exist in the segmentedregions which have been labeled as non-background regions. The featureextraction processing block 645 attempts to quantitatively describe anobject in a region. Once the regions contained in an image have beensegmented, statistics are accumulated which numerically characterizeeach region. As with the segmentation process, specific features for thegiven application must be derived.

The first feature that is calculated to help distinguish the objects ofinterest from the clutter in the image is the size of the regionscontaining the objects. A broad size evaluation is performed for eachregion to achieve a very general size elimination procedure. First apixel count is performed for each region. Next, the area of a pixel iscalculated for the image at the third resolution. The area of eachregion is then computed and compared to a maximum and a minimum value.If a region is too small or too large, it is merged into the background.Seldom will any object be rejected as being too big, but many smallclutter objects are eliminated. The maximum and minimum region sizes arebased upon the image resolution. The maximum region size employed canbe, e.g., 40 million per area of a pixel, and the minimum region size is20,000 per area of a pixel. The regions that pass the broad sizeevaluation are then analyzed further.

It is at this point that the processing performed to identify cloudsdiverges from the process performed to identify manmade objects. Toidentify manmade objects, the statistical features used are averagegradient magnitude, standard deviation of gradient magnitude, and area,perimeter, width and length of the region. In addition to these, threeother features are also calculated. The first determines how manyregions were originally in the segment before the adjacent regions weremerged. This feature can help distinguish between cultivatedagriculture, which will have many regions comprising the segment, and alarge singular manmade item such as an airfield or road, which will havefewer regions on the segment. The next two features count the number ofparallel lines and collinear lines that overlay each segment. Thesefeatures are also important in discerning between agriculture androads/buildings. After these features are calculated, the next step isperformed in the classification processing block 650.

Continuing with the processing to identify manmade activity, thedefinition of an object for this processing is one of four classes:road, structure, agriculture, or other manmade activity. Hence, theclassifier separates the manmade objects from the clutter. To reducefalse alarms, the classifier also identifies scan lines and any othersensor anomalies. Each region that passes the broad size evaluation ispassed through a rule-based classifier that determines one of theclasses of scan line, agriculture, road, structure, other manmadeactivity, or clutter. The rules are executed in the following order foreach region.

First the scan line set is called. The average gradient magnitude of theregion is compared to the average gradient magnitude of the image, andthe standard deviation of the gradient magnitudes is examined for eachregion. Agriculture rules are invoked next. These rules measure the sizeof the region, the number of parallel lines in the region, the number ofregions originally comprising the region, and the number of collinearlines in the region. If the object is not agriculture, then the rulesthat determine linear activity are employed. These rules use theregion's area, the region's length to width ratio, the number ofparallel and collinear lines in the region, and the number of regions inthe image.

The last set of rules to be invoked are the building rules. These rulesemploy the area, the length, the width, the perimeter, the length towidth ratio, and the number of parallel lines to classify regions. Theobjects that are not assigned a class by any of these rules default toclutter. Accordingly, the output of the classification processing block650 is an image with identified regions of manmade activity, theseidentified regions each containing an object.

Returning now to the feature extraction processing block 645, thesubsequent processing for identification of clouds will now bedescribed. To identify clouds, the statistical features which areemployed are average gradient magnitude, standard deviation of gradientmagnitude, area, perimeter, width, and length. In addition to thesethree other features are calculated. The first determines how manyregions were originally in the region before the adjacent regions weremerged. The next two features count the number of parallel lines andcollinear lines that overlay each region. After these features arecalculated, the next step is classification.

A cloud threshold is calculated based upon the original image terrainwhich can be determined a priori, e.g., either by terrain identificationprocessing block 215 or via cartographic information 210, whether snowis likely (determined a priori and from checking the image maximum andaverage intensities of the original image), the average intensity andthe standard deviation of the original image. This cloud threshold isused here and in subsequent cloud detection modules in the determinationand classification of cloud regions. To reduce the cloud false alarmrate, bright objects that are not clouds, such as snow, ice, orreflection on bodies of water, are classified into the category ofsnow/ice. The classifier also marks scan lines, which are linear imageanomalies caused by sensor scanning. Each segment that passed the broadsize evaluation is passed through a rule-based classifier thatdetermines one of the classes of: scan line; cloud; snow/ice; or other,i.e., non-cloud. The rules are executed in the following order for eachsegment.

First, the scan line set is called, followed by the snow/ice rule set.For both sets, the average gradient magnitude of the region is comparedto the average gradient magnitude of the image and the standarddeviation of gradient magnitudes is examined for each region. Forsnow/ice, these bright objects usually have average gradient magnitudesthat are much greater than the image's mean gradient magnitude, and thedeviation is often high. The next set of rules invoked are those whichclassify clouds. These rules also examine the average gradient magnitudeand standard deviation of each region, and determine if the shape of aregion is cloud-like, i.e., non-rectangular. The absence of linearstructure is also required by the cloud rules.

While the majority of clouds are segmented and classified correctly bythe processing described above, the cloud banks may not always besegmented. Cloud banks are larger areas of total cloud coverage. Cloudbanks are identified as areas of high gradient magnitude and are usuallyin the busy mask as separate regions. FIG. 24 illustrates the processingperformed by cloud bank identification block 655. To identify cloudbanks, the likelihood of their existence is checked by examining certainimage statistics. These image statistics include the standard deviationof gradient magnitudes for the entire image at the third resolution, themaximum gradient magnitude for the entire image at the third resolution,and the range of gradient magnitudes (max-min) for the entire image atthe third resolution. If the standard deviation, e.g., greater than 100,and the maximum gradient magnitude is large, e.g., greater than 900, andthe gradient magnitude range is wide, e.g., the difference between themaximum and minimum gradient magnitude values is greater than 500, thenthe cloud bank module continues. If these conditions are not met, thenthe processing of the image by the cloud bank identification block 655is complete and further processing is performed by cloud refinementblock 660.

If the cloud bank is expected (“YES” path of decision step 2410), thenthe snow/ice found by the classification module is masked in (step 2420)to ensure that they will not be identified as a cloud bank. Next, theaverage gradient magnitude of each region in the busy mask is comparedto the cloud threshold. Those regions that have a gradient magnitudevalue greater than the threshold are considered cloud banks and passedinto the cloud (“YES” path out of decision step 2430). Those regionswhich do not have an intensity value greater than the threshold areidentified as not being cloud banks and the processing for thesesegments of the image ends.

The purpose of the cloud refinement module is to provide a more accurateboundary around the cloud/cloud banks, and to detect new cloud regionsthat were not segmented, and therefore, could not be classified asclouds. FIG. 25 illustrates a functional flow of the cloud refinementprocessing block 660. The first step is the cloud boundary analysis. Theinput to the cloud boundary analysis processing block 2505 is theclassification image which has any objects found by other classifiers,all clouds and snow/ice identified in the classification module, and anycloud banks that were found. All pixels in the original downsampledimage at a third resolution that correspond to any pixels in theclassification image that are labeled cloud or cloud bank are checked tosee if their intensity values are above the cloud threshold. If theintensity values are not above the threshold, then the pixels in theclassification image are set to zero. After this deletion of pixels, itis possible that regions which are too small to be clouds are stilllabeled as clouds. Next, any cloud region less than 11 pixels aredeleted. In processing block 2510, new potential cloud pixels aredetermined. Specifically, the identification of any clouds that were notsegmented/classified, or were not part of a cloud bank are identified.All pixels that have not been classified are examined to determinewhether they are above the cloud threshold, with the requirement that nolinear structure is under them. A new possible cloud region image isgenerated from those pixels that meet these requirements. These newpossible cloud regions are uniquely labeled by processing block 2515.

Next, any region that is adjacent to an object is removed and any newregion that is adjacent to a cloud is labeled as a cloud by processingblocks 2520 and 2525. Adjacency can be defined as a 3×3 neighborhood.All remaining regions are then checked for sufficient size, e.g.,greater than 10 pixels, and small regions are deleted in processingblock 2530.

Next, the regions are examined to determine if they have cloud-likecharacteristics by processing block 2535. The rules used for thisdecision process determines if each region is a possible cloud byexamining the mean intensity of the region, the number of linearfeatures and magnitude pixels under the region, and the area. Allregions that meet the criteria are labeled and output as a separateimage. This image, along with the updated classification image, are thenpassed to feature extraction processing block 2540. Feature extractionprocessing block 2540 again checks the features on the new, as well asthe updated cloud regions, to ensure that they are still cloud-like. Thelast step is performed by processing block 2540 to classify all the newor updated regions into cloud or snow/ice. The image terrain type, thelikelihood of snow, the cloud threshold, the region size, the regionaverage intensity and intensity standard deviation (for the originalimage at a third resolution processed blocks 2510-2540), averagegradient magnitude and gradient magnitude standard deviation (for theupdated classification image), and a gray level co-occurrence featureare used to classify the regions and generate the final cloud image.

Bodies of Water

This portion of the processing identifies bodies of water, and bright ordark non-water regions in visual imagery. The bodies of water caninclude a river, lake or ocean. In addition to identifying bodies ofwater, the present invention also identifies the banks or edges of thewater as being distinct from the body of water. The techniques of thepresent invention can process extremely large images corresponding tohundreds of mega-pixels. The present invention employs the completepixel dynamic range in the identification of bodies of water, and inparticular employs a multi-resolution approach which outputs two images,one at each resolution. These outputs will be herein referred to asriver mask images.

In the multi-resolution approach of the present invention, the coarseresolution imagery, i.e., the original image downsampled to a thirdresolution, will be between 16 and 32 meters, while the mediumresolution imagery, i.e., the original image downsampled to a secondresolution, is between 4 and 8 meters resolution. FIG. 26 illustratesthe large scale processing performed to identify bodies of water. If theoriginal image is not at the proper resolution then the image will bedownsampled as illustrated in processing block 2610. Since theprocessing illustrated in FIG. 26 operates using large scale processing,the original resolution image will be downsampled twice to achieve animage at a third resolution, i.e., the coarse resolution image. It willbe recognized that the downsampling performed can employ the sametechniques described above in connection with FIGS. 7A-7D. In addition,although FIG. 26 illustrates a separate downsampling processing block,the downsampled image produced by the cloud identification processingmodule in processing block 610 can be employed instead of performing anew downsampling on the original image to produce the coarse resolutionimage at a third resolution. Using the downsampled image parallel edgesare identified by processing block 2620.

FIG. 27 illustrates the processing performed by the find parallel edgesprocessing block 2620. The downsampled image at a third resolution isprocessed by Kirsch edge detector processing block 2710 to identifyedges, i.e., areas of the image which border regions of differentintensities, in the image at a third resolution. Processing block 2720thins the identified edges to single pixel line segments. Nextprocessing block 2730 applies a threshold to remove weak lines, i.e.,lines which have small gradient magnitude values, from the thinned edgeimage. The threshold can be set such that, for example, the top 50% ofthe lines are retained. Processing block 2740 uniquely labels the linesremaining after thresholding and identifies end points for each labeledline. Processing block 2750 records the X, Y coordinates of each pixelon each labeled line.

Processing block 2760 is invoked to determine which edges are parallel.For each labeled line, each pixel on the line is compared to all thepixels on the other lines. Statistics are calculated to determine if twolabeled lines are parallel. To determine if two labeled lines areparallel, each labeled line's starting pixel is compared to all of theother lines pixels to find the minimum distance. This minimum distancepixel of the second line is then compared to every pixel of the firstline to find the second minimum distance. If these two distances do notdiffer by more than 10%, the starting points have been found, and thefirst set of end points of the parallel portion of the line pair isstored. This same technique is then employed to find the second set ofend points using the labeled line's ending pixel instead of the startingpixel. The two sets of end points are then checked to see if they cross,and then the processing block 2770 computes statistics. The statisticsare the average of the distance between the two lines, the standarddeviation of the distance change, and the length of the parallel matchof each line pair. The statistics are used to determine whether aparallel line pair should be considered further as part of a riversegment. Accordingly, processing block 2780 applies a threshold, basedupon collected statistics, to the line pairs. The threshold can be basedon the average distance and standard deviation of change, and isselected such that the smaller the average distance and the standarddeviation of change are for the parallel lines, the more likely thispair is retained. After thresholding, a parallel edge segment image isthen output to processing block 2630.

To identify the peaks and valleys, processing block 2630 generates anintensity histogram of the original image at a third resolution. Fromthe intensity histogram the peaks and/or valley points are selected. Itwill be recognized that the peaks of an intensity histogram are thoseintensity values which are associated with a large number of pixels,while the valley is the intensity value associated with the fewestnumber of pixels. Processing block 2640 selects an upper and lowerthreshold at the peaks and/or valley points to separate the image intotwo distinct images, a bright image and a dark image. The bright anddark image are then processed separately.

The dark image is processed by processing block 2650. Initially, in theimage output from the find parallel edges processing block 2620, theparallel line pairs are connected and the regions inside the connectedparallel line pairs are filled, and then labeled. Each pixel within theregion is checked to determine if it overlaps a non-zero pixel in thedark image. A tally of the number of pixels that overlaps with the darkimage is counted for each region. A result image of dark processing isgenerated. Those regions that have at least 100 pixels tallied arewritten into the dark result image as dark water regions. A border isadded around these regions. The remaining regions are written into thedark result image as dark, non-river regions. Any region that is in thedark image that does not overlap with the filled parallel line regionsis also written into the dark result image as dark non-river region. Aborder is also added around these regions.

Processing block 2660 processes the bright image. Initially, in theimage output from the find parallel edges processing block 2620, theparallel line pairs are connected and the regions inside are filled andthen labeled. Each pixel within the region is checked to determine if itoverlaps a non-zero pixel in the bright image. For each region a tallyof the number of pixels that overlaps with the bright image is counted.A result image of bright processing is generated. Those regions thathave at least 100 pixels tallied are written into the bright resultimage as bright river regions, and a border is added around thoseregions. The remaining regions are written into the bright result imageas bright non-river regions. Any region that is in the bright image thatdoes not overlap with the filled parallel line regions is also writteninto the bright result image as bright, non-river region. A border isadded around these regions. Finally, the results of the dark and brightprocessing are combined in processing block 2670 to produce a thirdimage identifying bodies of water in the image at a third resolution,thus ending the processing performed on the coarse resolution image.

FIG. 28 illustrates the medium resolution processing. The mediumresolution processing employs two primary features to distinguish waterareas. The first feature is the intensity variance, which should besmall between adjacent pixels in water areas in this type of resolutionimage. Second, the pixel intensity values in water areas are generallydarker than the background in the image, except with frozen water orhigh glare conditions. Accordingly, the medium resolution processingidentifies bodies of water by isolating dark pixel regions that havevery low variance.

If the original resolution image exists, and if a medium resolutionimage has not already been produced, processing block 2805 downsamplesthe original resolution image to a medium resolution image at a secondresolution. Next processing block 2810 removes scan lines. Since oneprimary feature measurement employed in the medium resolution processingis low variance, it is desirable to remove anomalies caused by scanlines in the intensity image at a second resolution. Specifically, scanlines tend to have high variance and may cause identified water regionsto separate. To correct for this anomaly, processing block 2810identifies groups of pixels that have a Δ between intensity values ofadjacent pixels that is less than 5. If such a group of pixels exists,the processing block then looks in the next column of the image at asecond resolution to see if there is a similar group of pixels that havea Δ of intensity values that is also less than 5. Next an averagedifference of intensity values across the columns is determined, andthis average difference of intensity values is employed as a Δcorrection for that column. To correct the data for a particular column,all of the Δ's for all previous columns are summed up to, and including,the current column, and each pixel's value is modified in that column bythe Δ correction amount. This corrected intensity image is employed inall further processing at the medium resolution to identify bodies ofwater.

Processing block 2815 applies a variance filter to the image output byprocessing block 2810. FIG. 29A illustrates the filter employed byprocessing block 2815 and FIG. 29B illustrates how to calculate thevalues for the filter. The filter essentially measures the intensityvariations from the mean intensity of the pixels. After the variancefilter has been applied, a threshold is applied by processing block 2820to remove any pixels in the variance output whose value is greater than10. Processing block 2820 then cleans the variance output image using aspatial filter to perform a series of erosions followed by a series ofdilations. The use of a spatial filter to perform erosions and dilationsis described in more detail below in connection with FIGS. 33A and 33B.Processing block 2825 labels every region with a unique value anddetermines the number of pixels in each region. Any region smaller than100 pixels is removed.

Processing block 2830 calculates histograms of both the variance imageoutput by processing block 2825 and the corrected intensity image outputby processing block 2810. Processing block 2830 next cleans the labeledvariance image. To clean the labeled variance image it is first expandedtwice. Specifically, the expansion involves changing the value of everyzero pixel that has a non-zero neighbor to its neighbor's value. Next,the expanded image is shrunk twice. Shrinking is the reverse of theexpansion process. The expansion and shrinkage fills in the holes in thelabeled regions, as well as smooths the borders of the regions.

Processing block 2835 calculates features for each labeled region in thecleaned variance image. The first set of features is density change,edge change and edge change percent. These features are selected becausewater tends to have a uniform variance and smooth edges. The densitychange feature measures the amount of holes in the region. The edgechange is a measure of the angle change going from one pixel to the nextalong the outline of the region. The edge change percent is the edgechange value from the original to the cleaned variance image.

The next set of features which are calculated include edge strength andintensity mean. Due to the change in intensity going from water to land,the edge strength along the water is usually high. The edge strengthfeature is determined using the average edge strength for the entireregion border. In addition, since the pixel values of the intensityimage for water are low, another feature which is calculated is theintensity mean for each labeled region. Accordingly, processing block2835 outputs the features of intensity mean, edge strength, densitychange, edge change, and edge change percent for each labeled region.Processing block 2830 outputs secondary features of variance mean,variance sigma, intensity image mean, and intensity image sigma for theentire image. These features are employed by water classificationprocessing block 2840 to determine which regions are water. Of thoseregions which identified as water, the ones that are most river-like areidentified by processing block 2845 and labeled by this processing blockas a river. To classify bodies of water as a river, the length and widthof the region is employed. Specifically, a ratio of length-to-width isemployed. Since rivers are typically much longer than they are wide, alarger value of this ratio is more likely to be a river. Next, the waterboundaries are identified by processing block 2850 which outputs animage with pixels labeled as water or river.

Small Scale Processing

As illustrated in FIG. 2, there are four types of processing whichemploy the original resolution imagery, this type of processing beingcollectively referred to as small scale processing. These four types ofprocessing include small object detection processing, linear objectdetection processing, small scale lines of communication processing, andbuilding/structure detection processing. Detection of lines ofcommunication is performed using both midscale (images at a secondresolution) and small scale (images at the first resolution) resolutionimagery. Hence, the small scale lines of communication processing willbe described in connection with midscale lines of communicationprocessing. FIG. 30 illustrates the processing components employed forsmall scale processing except for small scale lines of communicationprocessing. Not all of the processing blocks illustrated in FIG. 30 areemployed in each of the small scale processing. However, the particularprocessing blocks employed in each of the different types of small scaleprocessing will be identified below in connection with the descriptionof such processing, and duplicate descriptions of the processing blockswhich are employed in more than one of the small scale processing typesare omitted in the description below.

As illustrated in FIG. 30, small scale information 3005 and a prioriinformation 3010 are provided to an edge extraction processing block3015. The edge extraction processing block 3015 provides information toterrain identification processing block 3020. Terrain identificationprocessing block 3020 also receives large scale and mid-scaleinformation 3025 from the processing performed on the downsampledversions of the original resolution image. The terrain identificationprocessing block provides outputs to edge analysis processing block 3030and morphological filter 3035. Edge analysis processing block 3030 andmorphological filter 3035 provide outputs to small object classificationprocessing block 3040, which in turn, provides an output to structurenomination processing block 3045. Edge analysis processing block 3030also provides an output to line extraction processing block 3050. Lineextraction processing block provides an output to line relationshipprocessing block 3055. Line relationship processing block 3055 providesan output to rectangular object identification processing block 3060 andto linear object identification processing block 3065. Rectangularobject identification processing block 3060 provides an output tostructure nomination processing block 3045. Linear object identificationprocessing block 3065 provides an output to linear object nominationprocessing block 3070. Now that an overview of the various processingblocks employed in the small scale processing have been generallydescribed, a detailed description of each of these processing blocksfollows in connection with each type of small scale processing.

Small Object Detection

The present invention employs small scale information 3005, a prioriinformation 3010, edge extraction processing block 3015, terrainidentification processing block 3020, large/mid scale information 3025,edge analysis processing block 3030, morphological filter processingblock 3035, small object classification processing block 3040, andstructure nomination processing block 3045.

Edge extraction processing block 3015 processes the original resolutionimage to identify the edge direction, edge magnitude, local intensitymean. The edge directions and edge magnitudes are computed at everypixel. The edge extraction can be performed using a Sobel operator inthe manner described above in processing block 805 of FIG. 8 to producea gradient magnitude image. The local intensity mean is computed byconvolving the image with a 16×16 kernel of all 1's and dividing theresult by 256. The gradient magnitude local mean is obtained in asimilar manner, where the gradient magnitude image is employed insteadof the original intensity image. The gradient magnitude local mean imageprovides a good measure of the image busyness. This image describes thetexture or activity of an image, object or pixel. The greater themagnitude, the busier the texture around a pixel. Edge extractioninformation 3015 is provided to terrain identification processing block3020. Terrain identification processing block is employed to identifyterrain types in the image. As will be described in more detail below,depending upon the particular terrain identified in portions of theimage, different algorithms and thresholds will be employed fordetecting objects.

The small object detection processing employs morphological filter 3035to generate a filtered image identifying compact bright and darkregions. These regions are selected as potential objects, and the edgeanalysis processing provides additional information for identifyingsmall objects in the image. The small object detection processing isdesigned to detect objects which are less than 8 pixels by 8 pixels inthe original resolution image. The small object detection processingwill also detect larger objects, since medium and some large objects mayappear small in the gradient magnitude produced by edge extractionprocessing block 3015 due to noise or obscuration. It should berecognized that since small objects do not contain many pixels, theseobjects are harder to extract edge information from. The larger theobjects, the more edge information can be extracted from them to ensureit is a man-made object being detected, and not clutter. It has beenrecognized that since the edge information cannot be used reliably byitself, other features are also employed to detect small objects andthen classify them.

FIG. 31 illustrates the processing performed by terrain identificationprocessing block 3020. The terrain identification processing blockproduces an image mask which identifies terrain types in the image.Since terrain types can change at pixel boundaries in a particularimage, this image mask identifies terrain down to the pixel level. Thisterrain mask image will be employed in later processing to determineparticular threshold settings, and particular algorithms which should beemployed for different portions of the image. The present inventionemploys three types of information to generate the terrain mask image.The first type of information is global information about the image thatis determined by image parameters and statistics obtained by examiningthe entire image. Since a priori information may exist as to whichportion of the world is in the image, this a priori information can beemployed in generating the terrain mask image. The second type ofinformation employed is mask images generated from the large scale andmid scale processing. Specifically, the large scale cloud mask, and thelarge and mid scale river masks are employed to generate the terrainidentification mask image. The third type of information employed isterrain information which is determined in the terrain identificationprocessing block by the analysis of both the local intensity mean andthe local gradient magnitude mean for portions of the image. Forexample, the local values can be determined for portions which are 256pixel areas.

FIG. 31 illustrates the processing performed by terrain identificationprocessing block 3020. Initially, a local intensity mean histogram isgenerated (step 3105). Next, the local intensity mean histogram isexamined to determine if there are any valleys which exist in thehistogram (step 3110). If there are no valleys in the local intensitymean histogram (“NO” path out of decision step 3110), then the localgradient magnitude mean histogram is examined (step 3115). Next it isdetermined if there are any valleys in the local gradient magnitude meanhistogram (step 3120). If no valleys were present in the local gradientmagnitude mean histogram (“NO” path out of decision step 3120), thenthresholds from the large scale processing are employed to classify thepixel terrain (step 3125). If there were valleys in the gradientmagnitude mean histogram (“YES” path out of decision step 3110), or ifthere were any valleys identified in the local gradient magnitude meanhistogram (“YES” path out of decision step 3120), or after the largescale processing intensity thresholds have been identified (step 3125),then the image is segmented into bright and dark regions (step 3130).The region segmentation step 3130 employs either the valleys identifiedin the histogram, or the large scale intensity thresholds to segment theimage into regions. The segmented regions must be greater than, forexample, 1024 pixels to be retained. The image with the segmentedregions is then employed to identify the terrain type in the differentregions of the image (step 3135). As illustrated in FIG. 31, large andmid scale cloud and river masks are employed in the determination ofterrain type in different regions of the image. These large and midscale cloud and river masks will take precedence in the identificationof terrain in the image over any other decision with respect toclassification of the terrain in the image.

Most of the small objects in an image will be much brighter or darkerthan their immediate background. Accordingly, to detect small bright ordark objects, the present invention employs a morphological filter 3035.The morphological filter will identify potential objects that are lessthan the filter's equivalent radius and which are different inbrightness than the potential object's surrounding background. FIG. 32illustrates an exemplary morphological filter. As illustrated in FIG.32, an input image 3205, i.e., the image produced by edge extractionprocessing block 3015, is provided to a spatial filter 3210. The inputimage is also provided to subtraction processing block 3215. Subtractionprocessing block 3215 also receives a spatially filtered image fromspatial filter 3210. Subtraction processing block 3215 subtracts thespatially filtered image from the input image, and outputs a high passfiltered image identifying potential objects which have a smaller radiusthan the size of the filter and a different brightness than backgroundpixels surrounding the potential objects.

As illustrated in FIG. 32, the spatial filter employs a series ofdilations and erosions. FIG. 33A illustrates the components employed inthe dilation processing, while FIG. 33B illustrates the components ofthe erosion processing. As illustrated in FIG. 33A, a dilation operationinvolves convolving a 3×3 portion of the input image with a 3×3 dilationneighborhood containing all one values. As illustrated in FIG. 33B, theerosion operation involves a deconvolution of a 3×3 portion of the inputimage with an erosion neighborhood of all ones. The dilation and erosionperformed by the spatial filter essentially acts as a low pass filter inthe space domain. Specifically, the spatial filter removes regions oflimited spatial extent less than the size of the composite filter. Thenumber of successive dilates or erodes determines the size and pixelsthat are effected by the operator. The use of a series of erosions anddilations tends to “wash away” or “fill” any small intensity cracks orspikes, leaving only the low frequency information.

Prior to providing an output to small object classification processingblock 3040, morphological filter 3035 thresholds the high pass imagefrom subtraction processing module 3215 for bright and dark objects.Since the present invention can accommodate various terrain types, thepresent invention employs a dynamic and a static threshold. The dynamicthreshold is based on 0.8% of the maximum and minimum intensityhistogram values produced by the morphological filter. Accordingly, ifthe image is bland, then there is very little difference in intensitybetween objects and the background, resulting in a high clutterextraction. To prevent this from happening a fixed threshold isemployed. The fixed threshold sets a minimum intensity value for brightobjects to pass to the output image of morphological filter 3035, and amaximum intensity value for dark objects to pass on to the output ofmorphological filter 3035. The fixed threshold values will varydepending on the type of terrain being processed. Accordingly, an imagewhich possesses, for example, four different type of terrain, may employfour different fixed thresholds, each different fixed thresholdemploying a minimum intensity value for bright objects and a maximumintensity value for dark objects.

The present invention employs a Kirsch edge operator for the edgeanalysis performed by edge analysis processing block 3030. The Kirschedge operator outputs edges in one of eight directions, every forty-fivedegrees. As illustrated in FIG. 34A, the Kirsch edge operator employsall four templates for convolution with each pixel's neighborhood. Thetemplate that produces the maximum magnitude is selected. Next, the signof this maximum magnitude is examined using the table in FIG. 34B todetermine which entry the maximum magnitude matches. This tableidentifies the edge direction, and the absolute value of this maximummagnitude becomes the magnitude. For example, if Diagonal 1 produced themaximum magnitude of negative 231, since its value is negative, the edgedirection for this pixel is 6 and the magnitude of the edge is 231.

The output of edge analysis processing block 3030 and morphologicalfilter 3035 are employed in the small object classification processingblock 3040. Specifically, the output of edge analysis processing block3030 is thresholded and compared with the thresholded output of themorphological filter to perform further clutter rejection. The thresholdemployed on the output of the morphological filter 3035 has already beendescribed. The threshold applied to the output of edge analysisprocessing block 3030 are two different thresholds for the gradientmagnitudes of the edges. Both of these thresholds are dynamic. Onethreshold is based on 6.5 times the average local gradient magnitude,and the other threshold is based on 0.05% of the maximum histogram valueof the local gradient magnitude. The maximum histogram thresholdguarantees that the object is the maximum in the region, and the averagethreshold prevents clutter in a bland region from being extracted.Accordingly, if the object segmented by the morphological filtercontains a strong edge that has passed the gradient magnitude threshold,then the object is retained and stored as a small object in the smallobject mask image. If the object segmented by the morphological filterdoes not pass the gradient magnitude threshold, it is removed from thesmall object mask image.

Since different regions within an image have different textures andbrightnesses, the regions are processed separately. Specifically, thesmall object classification processing block processes all regionsidentified in the terrain identification processing block except forregions identified as cloud and river regions by the large scale and midscale processing. By employing separate thresholds for each regionidentified in the image, the present invention avoids having a thresholdwhich is too low for busy areas, such as vegetation or bright busyareas, which would result in a false detection. In addition, byemploying separate thresholds for different regions, a threshold whichis too high for bland areas, such as, shadow or bright bland areas, isprevented. A threshold which is too high for bland areas would lead tomissed detections. The small object detection processing block is theonly small scale processing which operates on the regions marked by thevegetation mask image to detect any small objects that may be surroundedby trees or bushes. Accordingly, the small object classificationprocessing block 3040 outputs an image mask identifying small objects inthe original resolution image to structure nomination processing black3045. The structure nomination processing block 3045 then employs thismask image in other processing.

Linear Object Detection

Linear object detection processing detects manmade linear objects suchas roads, paths, bridges, walls, plow lines, or any other manmade linearstructures. Referring now to FIG. 30, the linear object detectionemploys small scale information 3005, a priori information 3010, edgeextraction processing block 3015, terrain identification processingblock 3020, large/mid scale information 3025, edge analysis processingblock 3030, line extraction processing block 3050, line relationshipprocessing block 3055, linear object identification processing block3065, and roads/trails nomination processing block 3070. The processingfor primitive extraction processing block 3015 and terrainidentification processing block 3020, as well as edge analysisprocessing block 3030 have been described above, and hence, a detaileddescription of these processing blocks is omitted. The processingperformed by line extraction processing block 3050 is similar to thatdescribed above in connection with FIGS. 17-21. The main differencebeing that the minimum line length is 8 pixels for small scaleprocessing compared to 30 pixels for large scale processing.

The gradient magnitude and gradient direction images produced by edgeextraction processing block 3015 is received along with an originalresolution image by line extraction processing block 3050. The lineextraction processing block 3050 converts edges, i.e., areas of theimage which border regions of different intensities, into straight linesegments to obtain objects with a single straight edged line, instead ofseveral small lines. The line extraction process is described above inconnection with FIGS. 17-21.

Returning now to FIG. 30, line relationship processing block 3055receives an input from line extraction processing block 3050. Dependingon terrain type, thousands of line segments may be extracted per image.Since most of the objects identified using the original resolution imageare manmade objects which are linear or are rectangular in shape, thepresent invention employs the relationship of the identified lines toidentify objects. This processing involves the connection of collinearline segments into longer straight line segments, determining which linesegments are parallel (for linear object detections), and which linesegments are perpendicular (for rectangular object detections). Theremaining lines that have no relationships will be discarded.

FIG. 35 illustrates a method for determining whether line segments arecollinear, and hence, should be one single line segment instead of twosmaller line segments. Due to noise, obstructions, or otherinconsistency in extracting edges, line segments may have gaps wherethey should be continuous. These gaps may result in only a partialdetection of an object, the partial detection appearing as noise in theimage, and thus being discarded as clutter. Accordingly, the methodillustrated in FIG. 35 attempts to connect collinear line segments whichhave been disconnected due to noise, obstructions or otherinconsistencies in the image. The method in FIG. 35 will also reattachthe line segments that were purposely disconnected when curved lineswere broken up in the method illustrated in FIG. 17. The methodillustrated in FIG. 35 smooths and connects small line segments intolong continuous lines. As illustrated in FIG. 35, each line segment iscompared to every other line segment within the slope tolerancediscussed above in connection with the line segment table, i.e., 0 to179°. For each pair of lines there are several conditions that must bemet, if the pair of lines pass all of the tests, a new line compositesegment will replace the two lines in the line segment table.

The slope tolerance requirement will have been met since only lines inthe line segment table within the slope tolerance are compared to eachother. The line distance test step 3520 checks the gap between the twoline segments to determine whether it is within a predefined distance.This predefined distance must be no longer than the average of the twoline lengths. To prevent lines that overlap from being connected, adistance greater than zero is required.

The test slope tolerance step 3530 calculates an imaginary line from thefarthest endpoints of the two lines. The slope of the imaginary line iscompared with the slope of the two lines. The difference between theslope of the imaginary line and the original pair of lines must bewithin a few degrees to pass this test. Next, the line pairs are testedto determine whether they are anti-parallel or parallel in step 3540.This step prevents the combining of lines with opposite gradientdirections. A gradient direction is considered anti-parallel to anothergradient direction if it is pointed 180° or more in the oppositedirection. For example, a gradient direction of “2” is anti-parallel toa gradient direction of “6”; and a gradient direction of “1” isanti-parallel to an edge direction of “5”. Anti-parallel lines areusually found on opposite boundaries of long objects with some finitewidth, such as a road or a highway. If the lines are anti-parallel(“Yes” path out of decision step 3540), then the two lines fail thistest and the next line pair is considered. If, however, the lines aredetermined to be parallel (“No” path out of decision step 3540), thenthe line strength of the two line segments are tested at step 3550. Twolines will be recognized as related if their line strengths are similar.The line strengths of two lines will be considered similar if thedifference between strengths is not more than 25%. It should berecognized that the more similar the strengths are, the more relaxed thedistance requirement test may be. If two line segments pass the linestrength tests (“Yes” path out of decision step 3550), then the two linesegments are retained, and merged into a single line segment. Using theprocessing illustrated in the method of FIG. 32, a large number of linesare combined and weeded out of further processing.

FIG. 36 illustrates the processing performed to determine if two linesare parallel. Each line segment in the line segment table is compared toevery other line segment to determine whether there is a parallelrelationship between the lines. Initially, it is determined whether thetwo lines are parallel based on their slopes (step 3610). It will berecognized that two lines may in fact be parallel although, based ontheir slopes, the two lines would in fact eventually intercept. Thiswould be due to noise in the image, and is accounted for in step 3610 byemploying a slope tolerance instead of requiring an exact parallel sloperelationship. The longer the lines are, the stricter the slopetolerance.

The line distance test (step 3620) determines whether the two lines arewithin a predetermined distance of each other. This predetermineddistance will be based on the resolution of the imagery and the type ofobjects to be identified. One skilled in the art could easily calculatesuch a predetermined distance. Next, any overlap between two linesegments is tested to ensure that the overlap is a minimum length, thisminimum length being based on the image resolution (step 3630). In step3635, the terrain type of the two line pairs is determined and providedto step 3640. The two lines are discarded if they are not within thesame terrain type. Step 3640 tests whether the lines are anti-parallelor parallel. If the lines are anti-parallel or parallel (“Yes” path outof decision step 3640), then the gradient magnitude of the two linesegments is tested in step 3650. The gradient magnitude of each linemust be nearly the same for both lines to be considered parallel. Thisis intended to reduce some lines which are due to natural features ofthe terrain. The strength feature is computed by dividing the number ofpixels, which pass the gradient magnitude threshold, by the total pixellength of the line. If the strength feature of the two lines is within apredetermined amount of each other, then the two lines are labeled asmembers of a parallel line group (step 3660). If the two line segmentsfail any of these tests, they are not considered parallel.

The final processing performed by line relationship processing block3055 is grouping parallel line pairs. This grouping is illustrated inFIG. 37. This grouping of the lines into parallel line pairs isaccomplished by examining each parallel group and ranking by line size.The longest lines being the highest ranked. Accordingly, the longestline is determined (step 3720) and is then compared with every otherline in the parallel group to find the next longest and closest line(step 3730). A new parallel pair is created when the best match is found(step 3740). The remaining lines in the parallel group are processed thesame until all the lines have a best parallel match (steps 3710-3750).Once a line is a part of a parallel pair, it is no longer considered inthe grouping process (step 3750). If there is an odd number of lines inthe parallel group the last line is discarded (step 3760). Finally, acentral axis is determined for each line pair (step 3770).

After the relationships of the lines are identified by processing block3055, the linear objects are identified in processing block 3065 androads, trails, and the like, are nominated as identified objects byprocessing block 3070. The processing performed by blocks 3065 and 3070is illustrated in FIG. 38. As illustrated in FIG. 38, the central axisof a line pair identified as parallel is employed for determining acenter line, and the minimum and maximum X and Y coordinates for theline pair (step 3810). Next, a histogram of the slope of all of theparallel line pairs is calculated (step 3815), and a slope cluster countis determined (step 3820). The slope cluster count in the slopehistogram is employed for determining the line length threshold (step3825). Accordingly, if a large number of lines have similar slopes, thelength threshold is raised, whereas if the lines are not of similarslope, the length of threshold is lowered. If there are a large numberof lines with similar length, the length threshold is raised, since itis likely that these lines are natural features of the terrain. All ofthe lines are compared to the threshold (step 3830), and lines which arebelow the threshold are removed as insignificant, while lines which passthe threshold are passed to the next step. Next, a length histogram iscalculated for all of the remaining lines (step 3835), and the lengthcluster count is determined (step 3840). A line length threshold is thenset based upon the length cluster count is employed for setting anotherline length threshold (step 3845). The remaining lines which are belowthe new line length threshold are removed as insignificant. Theremaining lines are identified as significant, and are analyzed basedupon a confidence value. The confidence value can be based upon linelength and average edge strength. Accordingly, loner lines with highedge strength values have greater confidence values. This analysisoutputs an image identifying roads, weak roads, trails, and weak trails,thus ending the processing for identifying linear structures in theoriginal resolution image.

Building and Structure Detection

Referring again to FIG. 30, the processing performed for building andstructure detection includes processing blocks 3005, 3010, 3015, 3020,3025, 3030, 3050, 3055, 3060, and 3045. The processing performed inconnection with processing blocks 3005, 3010, 3015, 3020, 3025, and3030, have been described above in connection with small objectdetection. The processing performed in connection with blocks 3050 and3055 have been described above in connection with the identification oflinear objects. Since the details of these processing blocks have beendescribed above, a further description of these processing blocks isomitted.

Perpendicular lines are employed to detect buildings, structures andother manmade objects. These other manmade objects can includecompounds, agricultural fields, and some vehicles. To identify suchobjects, lines that are perpendicular and intersect within a certaindistance are extracted from the image. It should be recognized thatperpendicular in this context is not meant to be an exact 90° angle, butinstead is used to label a grouping of lines that can create a 90° anglein object space. When viewing objects at any oblique viewing angle, notperpendicular to a surface, the right angles of the surfaces in objectspace can project either greater or less than 90° in image space. Usingthe sensor pointing geometry and a flat earth assumption, the actualangle can be predicted. However, this prediction may not be useful dueto noise and error of the extraction routines. Accordingly, the presentinvention places a tolerance on the variation of an acceptable 90°angle, which can be, for example, a variation between 65 and 115° Thisvariation was determined to be acceptable through experimentation. Asillustrated in FIG. 39, two line segments, if extended, could beperpendicular to each other. To determine whether two lines which couldbe made perpendicular by extending the line segments could be part ofthe same rectangular object, the present invention employs threerequirements that these lines must meet. The first requirement is thatthe lines be perpendicular within the angle tolerance of 65 to 115°; thelines must have a specific gradient magnitude strength; and the twolines must be within a specific distance of each other.

The strength requirement is employed to reduce clutter from theresultant image. Most manmade rectangular objects, such as buildings,have very distinct, strong edges i.e., large gradient magnitudescompared to the area surrounding the edges. The strength requirementnecessitates that the line strength feature for both line segments beingcompared is greater than 90%. In other words, more than 90% of thepixels on the line segment have passed the gradient magnitude threshold.A line strength requirement of 90% is sufficient for the detection ofbuildings or the detection of strong agricultural objects. If it isdesired to detect weak agricultural objects, the percentage should belowered to 30%.

The distance requirement compares all lines that have passed the angletolerance and strength requirement to determine if they intersect, orwill intersect if they were extended. This distance between the actualline to the intersection must not be more than half of the actual linelength for either line, or greater than 12 pixels, whichever is less.Accordingly, referring again to FIG. 39, distance d1 must not be greaterthan half the length of the horizontal line, and must also be less than12 pixels for the line to be extended. Similarly, distance d2 must notbe greater than half the length of the vertical line, and less than 12pixels. If two lines are found to have a perpendicular relationship,then if necessary, the lines are extended to create the intersection,and reflected to create a closed region segment. This extension of thelines and reflection of the lines is illustrated in FIG. 40.

FIG. 41A illustrates lines extracted from an image from a top view of asimple perfect building. Lines 1 and 2 depict the shadow edges of thebuilding, while lines 3-6 depict the actual building walls. The buildingillustrated in FIG. 41A is an ideal case, however due to noise, onlypartial lines will probably be extracted from the original image. FIG.41B illustrates a more realistic illustration of the lines which wouldbe extracted from an image. In FIG. 41B, wall lines 3 and 5 were notextracted from the original image, while shadow lines 1 and 2, and walllines 4-6 were only partially extracted. Accordingly, as illustrated inFIG. 41C, wall line 4 will be extended to shadow line 1, and these lineswill be reflected to form an enclosed rectangular object.

To avoid connecting a building wall with a shadow line as illustrated inFIG. 41C, the present invention employs a structure confidence featureto reduce the likelihood that these types of lines will be connected.FIG. 42A illustrates two detected and two reflected lines. Asillustrated by the dashed region in FIG. 42B, a search for wall evidenceis performed by looking for lines which are parallel with the extractedlines and are within a certain distance of the parallel lines. The linesfound during the search for wall evidence will typically be smaller thanthe extracted lines, otherwise these lines would be extracted with theother lines. The certain distance is a distance which is consistent withthe dimensions of typical buildings. These confidence features will beemployed to help retain rectangular patterns and label them as buildingstructures. Confidence is kept track of by maintaining a count of thenumber of parallel lines that exist for each perpendicular relationship.The more complicated a building, the higher the parallel line count, andhence, the higher the confidence. Information from the segmentationapproach described above in connection with small object detection inthe form of segmentation is employed to increase the confidence of theseperpendicular relationships. The segments are tested to determine ifthey can be walls, shadows, or whole building structures. The presentinvention employs the contrast values of these segments to determine ifthey are shadows.

Rectangular manmade objects consist of buildings, compounds,agricultural fields and vehicles. To detect any rectangular objectexisting in the original resolution image, a new object segment iscreated from overlapping perpendicular segments. To create thesesegments, the background of the image is labeled and any perpendicularline segment touching the background is tagged as an outer line segment.Accordingly, as illustrated in FIG. 43A, a rectangular object whichoverlaps two other rectangular objects will be identified by the polygonillustrated in FIG. 43B, employing only the outside edges in thenon-overlapping regions of the rectangles.

Finally, as illustrated in FIG. 44, objects are nominated by employing aBayes classifier. The Bayes classifier employs various extractedfeatures to classify the identified objects. The first feature is thearea based on area per pixel. Area is employed to assist in thecategorization of the objects into different feature bins, small, mediumand compound, since different size objects have different features. Asecond feature is the average intensity. The average intensity isactually two intensity values, one for the intensity of a segment, theother the intensity of the immediate background in the image. A thirdfeature is the standard deviation of a whole segment and the immediatebackground in the image. A fourth feature is the average intensity ofthe whole image. A fifth feature is the gray contrast which measures thedelta between the segment intensity and the background intensity. Asixth feature is the gradient magnitude which measures the edge activityinside the segment, as opposed to just outside the segment. A seventhfeature is the average mean gradient magnitude which is a 16×16convolution of gradient magnitude values on a segment. An eighth featurewhich is extracted is the morphological filter statistics which includethe mean, standard deviation, minimum and maximum values from thisfilter. A ninth value is the structure confidence which is a count ofthe number of lines that are parallel to the perpendicular relationship.It should be recognized that the size of a structure will affect thiscount. A tenth feature is the texture energy filters, which is astandard set of 5×5 convolution filters that measure the texture patternin an image region.

Using the extracted features, the Bayes classifier categorizes theobjects into building, agricultural, or other manmade activity, andclutter. The buildings are further subdivided into small building,building and compound/building cluster. Since the terrain type is knowna priori, a separate classifier can be used. Area, brightness, edgestrength, and geometric structure can characterize buildings. A majorityof the clutter falls into two groups, vegetation and crevices. Twoclassifiers are employed for forest terrain types, and two classifiersfor arid terrain types. These classifiers are trained on different datasets. The output from each classifier can be used separately and theresults added. Alternatively, both classifiers must nominate a candidatesegment for it to be categorized as a building.

Lines of Communication

The present invention employs small scale resolution imagery, i.e., theoriginal first resolution image, and mid scale resolution imagery, i.e.,the first resolution image downsampled to a second resolution, for thedetection of lines of communication. Lines of communication are manmadestructures used for transmitting or transporting products such asinformation, electric power, gas, oil, fresh water, wastewater, andtransportation structures such as surfaced roads, dirt, railroads, andpaths. Lines of communication may appear to lack continuity in an image.For example, lines of communication buried pipe or transmission linesmay appear as cuts through the forest. Lines of communication maymanifest themselves as strong linear structures containing multiple linesegments in the imagery, or due to shadows or some other obscuration.Hence, the present invention attempts to bridge this loss ofinformation. Since lines of communication can have a variety of sizesand thicknesses from, for example, a dirt path to a multi-lanesuperhighway, the present invention employs both small scale and midscale resolution images for the detection of lines of communication.

FIG. 45 illustrates a generic process which is employed in both thesmall and mid scale resolution processing. Primitive analysis processing4510 is performed on an input image 4505, either at a small scale or midscale resolution. The primitive analysis processing 4510 processes theinput image at the pixel level to identify simple structures that areelements of lines of communication. The primitive analysis processing4510 outputs an image identifying structures in the input image. Theimage output by the primitive analysis processing 4510 is input toprimitive cleaning processing 4515 to prune and/or thin the primitives.In this processing step, primitive connectivity can be tested toeliminate extra elements. The primitive cleaning processing 4515 outputsan image to river and cloud mask processing step 4520. In the river andcloud mask processing step 4520, the image output by primitive cleaningprocessing step 4515 is masked with the river and cloud mask to preventprocessing of any primitives which are located in portions of the imageidentified as rivers and portions of the image identified as clouds.This image is output from river and cloud mask processing step 4520 toHough transform processing step 4525. The Hough transform processingstep 4525 tests large groupings of the identified primitives todetermine if they are to some degree co-linear. This processing stepwill group the processed primitives into larger line segments.

FIG. 46 illustrates the processing performed for the identification oflines of communication in the small scale imagery. The originalresolution image is input to morphological filter processing block 4610.The morphological filter processing block 4610 employs the morphologicalfilter described above in connection with FIGS. 32 and 33, and hence,for a detailed description of this processing, one should refer to thedescription above. The image output from morphological filter processingblock 4610 is input to threshold processing block 4620. The thresholdprocessing block 4620 employs an intensity histogram of the image outputfrom morphological filter processing block 4610 to compute a threshold.The threshold is based on a specified percentage of the intensityhistogram. The threshold should be selected with a bias value to allowboth dark and bright intensities in the imagery to be examined.

The thresholded image from processing block 4620 is input to processingblock 4630 where the image is compared with an image which identifiesregions which have either rivers or clouds. To avoid false detections,regions which have either rivers or clouds are not to be processed inthe identification of lines of communication. Accordingly, the intensityvalues for regions of the thresholded image which have rivers or cloudsis set to zero to avoid false nominations of lines of communicationbeing identified in these regions of the image. The image output fromprocessing block 4630 is input to Hough transform processing block 4640.

The Hough transform processing block 4640 is performed on the image toidentify all line segments of importance in the image. The Houghtransform maps an image into an alternate discrete space, where eachelement or cell indicates how many pixels in the image lie on a linewith a particular angle and location. FIG. 47A illustrates the mappingof a single point in image space into Hough space. A single point getsmapped from image space into Hough space as a sinusoidal curve. FIG. 47Billustrates a series of points being mapped into Hough space. Asillustrated in FIG. 47B, different sinusoidal curves result from eachpoint being mapped. It will be noticed that all the curves illustratedin FIG. 47B pass through a single point in Hough space. Accordingly,multiple points of a line in image space will result in multiplesinusoidal curves in Hough space passing through a single point.Therefore, a large number of sinusoidal curves all passing through thesame point in Hough space, correspond to long lines in image space.Since the mapping from image space to Hough space is reversible, onceeach point in image space is mapped into Hough space, the lines in imagespace can be determined by looking for high counts in Hough space. FIG.48 illustrates the mapping function for mapping from image space intoHough space. The Hough transform is performed on the image by processing128×128 pixel overlapping windows. The use of overlapping windowsimproves the detection rate by reducing edge effects caused by objectscrossing window borders, and also allows for the detection of smallerlines of communication or lines of communication that have a slightcurvature.

Returning now to FIG. 46, since a large number of possible outputs areselected by Hough transform processing block 4640, processing block 4650acts as a cleaning stage to limit the number of lines identified in theimage. FIG. 49 illustrates the processing of clutter rejectionprocessing block 4650 in more detail. The clutter rejection processingblock 4650 processes each line identified in the image output from Houghtransform processing block 4640. Initially, each line is compared to aninitial threshold in processing step 4905. In step 4905, it isdetermined whether there are more than a predetermined number of pixelson the line. One skilled in the art will recognize that thepredetermined number of pixels is selected to balance the number ofmissed detections verses false nominations. If there are not more than apredetermined number of pixels on the line, then the line is discardedin accordance with the processing step 4910. If, however, there are morethan a predetermined number of pixels on the line, then the regionsurrounding the line is extracted in processing step 4915. Next theportions of the extracted region to the left and the right of theidentified line are examined. If the identified line is a valid line ofcommunication, then the portions of the region to the left and right ofthe line, which represent the background, should appear similar, and theidentified line itself should appear different from both the left andright regions. Accordingly, in step 4920 a mean value and standarddeviation of intensities is calculated for the left, middle and rightportions of the extracted region.

In step 4925 the extracted region is segmented and each segment isuniquely labeled. The segmentation is accomplished by thresholding theextracted region with a threshold of the mean intensity minus half ofthe sigma of the intensity of the extracted region. The length, width,longest cord, and cord angle are calculated in accordance with steps4930 and 4935. It will be recognized that the cord is a line between theminimum X and Y pixel coordinate and the maximum X and Y pixelcoordinate. The cord angle is the angle with respect to the Hough line.To avoid being rejected as clutter, each line segment should have a cordangle which lines up with the center line, the cord end points of thesegment should be close to the center line, the segment should benarrow, the segment should be long, and the average distance of segmentpixels from the center line should be small. Each of the labeledsegments which satisfy these characteristics, has its length summed, andits mean width averaged. In step 4940 four rejection values are employedfor eliminating clutter. These rejection values are the mean width, thesum of the segment lengths, the sigma of the left and right portions ofthe extracted region, and the mean intensity of the left and rightportions of the extracted region. If these values pass the thresholds,then the line is considered an element of a line of communication. Sinceas the Hough processing is scrolled through the image, a single elementof a line of communication may possess several nominated lines, thefinal step of the processing is to eliminate multiple lines nominatingthe same element of the lines of communication.

FIG. 50 illustrates the processing performed in the detection of linesof communication in the mid-scale resolution image. Initially, the imagewith a second resolution is input to tuned filter processing step 5010.FIG. 51 illustrates the tuned filter employed in the present invention.The filter illustrated in FIG. 51 is applied to the image at the secondresolution. Then the filter is rotated through 22.5 degrees and appliedto the image at the second resolution again. This is repeated for allangles, and requires 16 rotations of the filter. The number of filterapplications can be reduced to 8 via symmetry, and hence, eightdifferent filtered images are produced. The results from each pass areoutput into one file. The output of the filter is employed as an inputto the clean and thinning processing step 5020.

FIG. 52 illustrates the cleaning filter which is employed to remove anysmall regions from the image. The image may contain long linear lineswhich are several pixels wide after cleaning. Since the Hough analysisrequires the lines to be one pixel wide, a thinning step is performed.The thinning step examines each pixel's neighbors, and removes the pixelif it does not break continuity of the region. FIG. 53 illustratesseveral examples of 3×3 pixel areas where the center pixel would be setto zero. Every eight connected pixel grouping in the thinned tunedfilter image is labeled with a unique value. A pixel is eight connectedif one of its immediate eight neighbors is also a member of the set.Recorded with this unique group value is the length and pixels of theline segment, and the mean and maximum value of all the members of theline segment. Since clutter regions tend to have all low values, whilelong linear objects have at least one or more large values, thresholdsare set on the length, mean intensity and maximum intensity values ofeach line segment to reject clutter in the image. The maximum valuethreshold is a function of the segment mean line. The larger the meanline segment, the larger the threshold of the maximum value. If themaximum value is less than the threshold, the line segment or region isremoved. All line segments with just a few pixels are also removed. Theremaining line segments are next checked to determine if they arelinear, thereby forming a straight line.

The cleaned and thinned image is provided to river and cloud maskprocessing block 5030 which outputs the cleaned and thinned image withall regions where rivers and clouds are identified with a value of zero.This image is input to Hough transform processing block 5040 which isused to determine if the line segment is linear or curved. Beforeapplying the Hough transform processing, each line segment is cleanedusing a dilate morphological filter. The dilate morphological filterchecks each pixel to determine whether it is a line segment pixel. Ifone of its neighbors is on the line segment, this pixel is converted toa line segment pixel. For each line segment a bounding box is determinedthat completely encloses the line segment. The Hough processing isperformed on the region contained within the bounding box. FIG. 54illustrates a bounding box in accordance with exemplary embodiments ofthe present invention. The Hough transform processing outputs the bestfit angle and the magnitude of the match or maximum value. The number ofpixels in the bounding box is divided into this maximum value to form apercentage “r”. If the count of the number of pixels in the labeledregion is greater than 100 and if the “r” is greater than 80%, it isdetermined that the line in the bounding box is an acceptable straightline. If the count is less than 100 pixels, then the followingrelationship is used to determine if it is an acceptable straight line:if r≧(1.0−0.002×count) then accept as straight line.

The Hough processing is next performed on the entire image rather thanjust the bounding box. The Hough transform processing is applied to theentire image to try to connect separate regions that lie in the sameHough transform space. The Hough transform processing is performed on512 pixel square windows with 256 pixel overlaps as illustrated in FIG.55. This process is performed enough times to cover the total size ofthe image. Again a threshold is placed on the number of counts in themaximum bin. If this threshold is not exceeded, the lines are discarded.If it is exceeded, the maximum tuned filter magnitude of the two regionsis summed, and the tuned filter mean is calculated for this maximum bin.Using the mean value, the maximum sum is tested against a threshold. Ifit passes the threshold, the line is identified as a mid-scale line ofcommunication. Since scan line artifacts may exist, the residual linescanned segments can be tested to determine if it is a true line segmentor a scan line artifact. All scan line artifacts are eliminated, and ifnecessary, are stored or are sent out as a flag.

Contextual Line Reasoning

FIG. 56 illustrates a high level block diagram of the contextual linereasoning processing. As illustrated in FIG. 56, the contextual linereasoning processing has inputs of large scale detection of activity,including roads, mid-scale lines of communication, small scale lines ofcommunication, and small scale linear objects such as roads. Thecontextual line reasoning processing uses these inputs to output animage identifying long linear objects. The purpose of the contextualline reasoning is to remove redundancy among the detection techniques sothat all of the nominations are fused into one unified set of detectionvectors. The contextual line reasoning also attempts to connectidentified objects from all of the linear detection techniques, forexample combining roads identified in large scale resolution imagery androads identified in small scale resolution imagery. The contextual linereasoning processing will also remove spurious detections that do notmeet physically realizable roads or lines of communication. Thesespurious detections may manifest themselves as zig-zag structures thatare detectable and can be eliminated. It will be recognized that in alldetection routines it is possible to miss sections of roads or lines ofcommunication. Accordingly, the contextual line reasoning provides theframework and data to improve connectivity by bridging gaps in roadsand/or lines of communication. However, contextual line reasoning willnot eliminate lines that can be used to detect agriculture or whichsupport urban city streets.

FIG. 57 illustrates the processing performed by the contextual linereasoning process. Initially, the outputs from each line processor aremerged in processing step 5705. Next the vectors are converted into theoriginal scale format so that all of the detections are mapped into acommon framework. Accordingly, the large scale images at a thirdresolution will be upsampled twice into the small scale images at firstresolution, and the mid-scale images at a second resolution will beupconverted one time to achieve the small scale images at a firstresolution. In processing step 5715 the primitive line segments areconnected.

FIG. 58 illustrates the processing steps to connect the primitive linesegments. Initially, in step 5805, the lines are stored as vectors of x,y coordinate pairs. The remainder of the processing measures theorientation of all of the line pairs. If a line pair has an includedangle that is less than 5°, (“True” path out of decision step 5815),then the line pairs are evaluated for connection in steps 5820 through5835. FIG. 59B illustrates the included angle. If the vectors intersect,if the vectors are less than 16 pixels apart, or if there is aseparation that is less than 25% of the vector length (steps 5820, 5825or 5830) then the line pairs are connected, and if necessary, a vectorfor connection is created in processing step 5835.

If the included angle is not less than 5° (“False” path out of decisionstep 5815), then it is determined if the included angle is less than 45°in processing step 5840. If the angle is less than 45° (“True” path outof decision step 5840) then the vectors are connected if they intersect(step 5845), or if there is a separation distance of less than 25% ofthe vector length (step 5850). If necessary, a vector is created forconnection of the two vectors in processing step 5835. If the angle isnot less than 45° (“False” path out of decision step 5840), then noconnection is made in accordance with processing step 5855.

All pairs of lines are tested in accordance with the processingillustrated in FIG. 58 and connected if they pass. Once a pair ofvectors are connected, the new vector is placed in the vector table andthe two component vectors are marked as invalid. Any vector created toperform the connection is also associated with the new vector, but areplaced in front of the base vector in the list since they are not to betreated as a created vector. Once all vectors have been scanned in thelist, the base vector is advanced one position and the process isrepeated until the base vector advances to the end of the list.

Returning now to FIG. 57, the connected primitive line segments areinput to density clutter rejection processing step 5720. FIG. 60illustrates in detail the processing performed in density clutterrejection processing step 5720. The density clutter rejection processingstep is intended to eliminate vector groups that are non-linear-like.Accordingly, a vector group is selected in processing step 6005. Usingthe selected vector group, a minimum enclosing box which entirelysurrounds the vector group is determined and the area of the minimumenclosing box is calculated in processing steps 6010 and 6015. Inparallel with step 6010, the number of pixels in all of the lines in thevector group is calculated in processing step 6020. In processing step6025 the area of the minimum enclosing box is divided by the number ofpixels in all of the lines in the vector group. The result of thecalculation in step 6025 is compared to determine whether it is lessthan 0.01. If the result of the calculation step 6025 is less than 0.01,then there is a low pixel to area ratio, and hence, it is likely thatthe vector group is in fact a linear detection. If, however, the resultof the calculation and processing step 6025 is not less than 0.01, thenthere is a high pixel to area ratio indicating that the vector grouptended to stay “balled up” and did not cover any appreciable distance aswould be the case in a road like structure. Accordingly, such a vectorgroup would be rejected as clutter.

Density clutter rejection is accomplished by processing a vector groupat a time. Each vector group is scanned for the minimum and maximum xand y coordinates. Once these values are obtained, a Δx and a Δy arecalculated. The maximum of these deltas is then used to define the areaof the box encompassing the vector group. Next each vector in the vectorgroup is scanned for their Δx and Δy. The maximum of these two deltasdefines the number of pixels comprising that vector. These vector pixelcounts are then summed, and the total is then divided by the area of thebox calculated earlier. This value is then tested against the thresholdvalue of 0.01, and rejected as clutter if greater.

Returning again to FIG. 57, after the density clutter rejectionprocessing step 5720, the remaining linear structures are processed inpath analysis processing step 5725. The path analysis processing step isintended to find the best path through the vector. The best path isdefined to be the longest, most direct path between any two end pointsin the vector group. This path usually occurs along the major axis ofthe vector group. This path is found by calculating the most directdistance between every possible pair of end points within a vectorgroup. These lengths are then scanned to find the two end points withthe longest path length. Once this path has been determined, all vectorsassociated with this vector group, but not involved in the path, aredeleted from the group. This results in a cleaner line segment.

In order to perform path analysis, a matrix of single path distances isformed for all the vectors in the group. This matrix is denoted as M¹,which is the single jump matrix, and represents the shortest distancefrom any node A to any node B in the vector using at most one vector. Asillustrated in FIG. 61, by a matrix operation of M¹ with itself, matrixM² can be produced. Matrix M² represents the shortest distance from anynode A to any node B in the vector group using at most two connectingvectors. Matrix M² is calculated by iterating through each node pair, Aand B, to determine the shortest distance using another node C. If thereis more than one node C where this is possible, the shortest distance isplaced in the location (A, B). Let {circle around (X)} be the shortestdistance operator, so M²=M¹{circle around (X)}M¹.

FIGS. 62A-62E illustrate the best path analysis. Specifically, FIG. 62Aillustrates the distance between a plurality of nodes 1 through 5. FIGS.62B through 62E respectively illustrate the single jump matrix throughthe four jump matrix of the nodes illustrated in FIG. 62A. Accordingly,in the matrix M¹ it can be seen that the 4,2 value in the matrix is 15,which represents the value of the length between nodes 2 and 4 in FIG.62A. It should be noted that the matrix is triangularly symmetric, sothat only the upper triangle needs to be filled in. Since matrix M³covers all possible connections between the nodes, as illustrated inFIG. 62A, matrix M³=M⁴. This equality determines the end of theprocessing. Since there can be relatively long paths contained in theinner nodes, it may require several iterations to solve all of thematrices. By performing the higher iteration count on a smaller matrixfirst, operations can be saved. Once the inner node matrix has beensolved, and since all end points are connected to inner nodes, it willtake two iterations of the full matrix to complete it.

Once the matrix has been completed, the distance from each node to allother nodes is known. Now the matrix is scanned to find the nodes thatare the farthest apart. These two nodes will always be end points. Oncethe two end points are found the correct path must be traced. Firstchoosing one of the end points as the current node, then calculating thedistance to each neighbor node using the M¹ single node matrix plus thedistance from that node to the other end point using the solved matrix.Then proceed to the node which yields the shortest overall distance.Each current node is recorded which builds the path, and the operationis stopped upon reaching the other end point. Once the path is known allother vectors can be removed, generally rendering a cleaner outline ofthe identified linear object. It has been recognized thatM^(2n)=M^(n){circle around (X)}M^(n) the processing time can belogarithmically reduced. Arranging the interior nodes, the nodes withmore than one single path connection, first in the list further reducesprocessing time.

Returning again to FIG. 57, the paths which remain after the pathanalysis processing step 5725 are processed in zig-zag clutter rejectionprocessing step 5730. FIG. 63 illustrates the zig-zag clutter rejectionprocessing step 5730 in more detail. Initially a vector group isselected and compared to a path length threshold in processing steps6305 and 6310. The threshold can be, for example, 256 pixels in length.If the vector group has a path length which is less than the thresholdpath length then it is discarded. If, however, the selected vector grouphas a length exceeding the threshold than the number of angles greaterthan 90° is calculated in processing step 6315. The number of anglesgreater than 90° is divided by the length of the path in processing step6320. If the result of the division is less than 0.01, then it isdetermined that there are no zig-zag line segments and the vector groupis retained. If, however, the result of the division is greater than0.01, then it is determined that it is a zig-zag line segment and thevector group is rejected.

Returning again to FIG. 57, the lines which have passed the zig-zagclutter rejection processing step 5730 are provided to vector groupconnection processing step 5735. FIG. 64 illustrates, in more detail,the processing performed in step 5735. FIG. 65 illustrates theconnection of vector groups in accordance with exemplary embodiments ofthe present invention. At this point in the processing, vectors havebeen combined based on their orientation and location, all paths havebeen connected, large clutter groups have been removed and zig-zaggroups have been eliminated. The remaining linear structures are longgroups of linear structures. Accordingly, any breaks in these linearstructures should be connected based on distance and line orientation.Breaks are usually caused by obstruction from clouds, terrain masking,or possibly roads or trails that have low contrast relative to thebackground. Accordingly, all vector groups are compared and the closestend point is determined in processing steps 6405 and 6410. Next, inprocessing step 6415, all angles of the connection triangle are testedto determine if they are all less than 45°. Next it is determinedwhether the length of the connection between the vector groups is lessthan 25% of the maximum length of the two vectors in processing step6420. If the result of the tests in step 6415 and 6420 are successfulthen the vector groups are connected. If, however, one of the tests insteps 6415 and 6420 are not successful, then the vector groups are notconnected.

Returning again to FIG. 57, after the vector group connection processingstep 5735, a second path analysis and a second zig-zag clutter rejectionprocessing step are performed in processing steps 5740 and 5745. Theprocessing steps 5740 and 5745 are performed since clutter regions maybe created in the vector group connection processing step 5735. In theprocessing steps 5740 and 5745, these are performed similar to the firstpath analysis processing step 5725 and the first zig-zag clutterrejection processing step 5730, however, the thresholds are twice theoriginal values employed in the first processing steps. All lines orpaths that make it through this process are labeled an output from thecontextual line reasoning processing.

Scene Context Analysis

The present invention employs cultural paradigms to reduce falsedetections. Most of the detection processes described above employ localimage operations to identify manmade activity. These detections arelabeled with a confidence or strength measure that is expressednumerically. The confidences can vary from weak or low confidence tostrong or high confidence. The scene context analysis is performed totest the plausibility or existence of the weak confidence manmadeactivity detections. This process improves the detections orclassification performance without increasing the false selection ornominations. The scene context analysis is preferably employed in anexpert system employing rules which specify the organization orstructure of the plausible manmade activity relationship.

FIG. 66 illustrates the processing performed in the scene contextanalysis. The terrain type and geographical location are employed in theselection of the cultural paradigms for the scene context analysis.These cultural paradigms determine the specific manmade activity rulesemployed. Each of the detection processes described above employ a setof criteria or thresholds that determine if the detected target is of aparticular class type or is a clutter object. In the scene contextanalysis these final thresholds or criteria are removed or reduced,thereby providing more detections into the scene context analysisprocess. Using the determined relationship rules the various detectionsare tested for plausibility. If, based on the relationship rules, thedetection is identified as plausible, the detection is kept and theconfidence value associated therewith is raised. If, however, it isdetermined based on the relationship rules that the detection is notplausible, the detection is removed from any further processing.

It will be recognized that various relationship rules can be devisedbased upon various cultural paradigms. For example, a low confidencebuilding can be retained if it is a specific distance from a detectedroad or trail. The particular geographic location and terrain typedetermine the maximum distance criteria. Distances measured as theshortest perpendicular from the building to a road or trail. If thereare more than one road or trail running near this candidate building,the shortest distance road or trail is the one considered for this rule.Low confidence buildings may also be retained if they are a specificdistance from a high confidence building. Again, this relationship isdependent on the geographical location and the terrain type. Typically,small buildings are missed in the detection processing becausethresholds are set high to reduce the number of false nominations. Oncea high confidence building is detected, it is possible that there areother buildings in the area. Using this distance criteria assists inidentifying additional buildings in the area without significantlyincreasing the false nominations.

Although specific relationship rules have been discussed in connectionwith the scene context analysis, one skilled in the art will recognizethat rules other than those specifically discussed above can beimplemented. The selection of these rules is based on cultural paradigmsfor the specific portion of the world captured in the processed image.Employing cultural paradigms based on which portion of the world iscaptured in the image can reduce false identification of objects byaccounting for how local cultures may layout manmade objects. Forexample, in some parts of the world, buildings will always be locatednear roads, whereas in other parts of the world buildings may not belocated proximate to roads.

The present invention has been described with reference to severalexemplary embodiments. However, it will be readily apparent to thoseskilled in the art that it is possible to embody the invention inspecific forms other than those of the exemplary embodiments describedabove. This may be done without departing from the spirit of theinvention. These exemplary embodiments are merely illustrative andshould not be considered restrictive in any way. The scope of theinvention is given by the appended claims, rather than the precedingdescription, and all variations and equivalents which fall within therange of the claims are intended to be embraced therein.

1. A method for identifying linear objects in an image comprising:identifying, in a processor of a computer processing device, terraintypes in the image; generating, in the processor, a gradient vectorimage from the image, the gradient vector image identifying a gradientmagnitude value and a gradient direction value for each pixel of theimage; identifying, in the processor, lines in the gradient vector imageusing the identified terrain types in each portion of the image;determining, in the processor, whether the identified lines areperpendicular, collinear, or parallel to another line among theidentified lines in the gradient vector image; eliminating, in theprocessor, lines among the identified lines which are determined to notbe perpendicular, collinear, or parallel to another line among theidentified lines in the gradient vector image; and identifying, in theprocessor, linear objects using the remaining identified lines whichhave not been eliminated.
 2. The method of claim 1, wherein the step ofidentifying lines using the identified terrain types comprises: applyingthe identified terrain types to the gradient vector image; determining athreshold based on mean gradient magnitudes for each individualidentified terrain type; and eliminating pixels in the gradient vectorimage having a gradient magnitude not greater than the threshold for theidentified terrain type associated with the eliminated pixels.
 3. Themethod of claim 2, wherein the step of identifying lines furthercomprises: comparing each remaining pixel that has not been eliminatedin the gradient vector image to neighboring pixels; and eliminating aremaining pixel if the gradient magnitude value of the remaining pixelis less than the gradient magnitude value of one of the neighboringpixels.
 4. The method of claim 3, wherein a pixel is determined to be aneighboring pixel based on the gradient direction value of the remainingpixel.
 5. The method of claim 3, further comprising: grouping pixels ina line with the same gradient direction value into a region, wherein thegrouping accounts for errors in a respectively corresponding gradientdirection value of the grouped pixels.
 6. The method of claim 5, furthercomprising: dividing the region into a plurality of subregions if theregion is greater than a predetermined number of pixels; and determininga straight line approximation for at least one of the subregions,wherein the straight line approximation is based on a calculated slopeand center of mass for the at least one of the subregions.
 7. The methodof claim 6, further comprising: determining a slope, endpoints,Y-intercept, and length of each of the identified lines or straight lineapproximations; and storing the slope, endpoints, Y-intercept, andlength of each of the identified lines or straight line approximations.8. The method of claim 7, wherein the determination of whetheridentified lines are collinear or parallel to another line among theidentified lines in the gradient vector image comprises: comparing eachidentified line or determined straight line approximation to every otheridentified line or determined straight line approximation with a similarstored slope to determine (i) whether the identified line or determinedstraight line approximation is a predetermined distance from otheridentified lines or straight line approximation with a similar storedslope, (ii) whether the identified line or determined straight lineapproximation is antiparallel to the other identified lines ordetermined straight line approximations with a similar stored slope, and(iii) whether the identified line or determined straight lineapproximation and the other identified lines or determined straight lineapproximations with a similar stored slope possess gradient magnitudeswithin a predetermined tolerance.
 9. The method of claim 8, wherein thedetermination of whether identified lines are parallel to another lineamong the identified lines in the gradient vector image comprises:determining whether each identified line or determined straight lineapproximation is within the same terrain type as the other identifiedline or determined straight line approximation, wherein if the lines arenot within the same terrain type, the lines are determined not to beparallel.
 10. The method of claim 1, further comprising: groupingidentified lines determined to be parallel into a parallel line group;and dividing identified lines in each parallel line group into parallelline pairs based on a respective length of the lines.
 11. The method ofclaim 10, wherein the linear objects are identified using the parallelline pairs.
 12. A non-transitory computer-readable recording mediumhaving a computer program recorded thereon that causes a computer toautomatically identify linear objects in an image, the program causingthe computer to perform operations comprising: identifying terrain typesin the image; generating a gradient vector image from the image, thegradient vector image identifying a gradient magnitude value and agradient direction value for each pixel of the image; identifying linesin the gradient vector image using the identified terrain types in eachportion of the image; determining whether the identified lines areperpendicular, collinear, or parallel to another line among theidentified lines in the gradient vector image; eliminating lines amongthe identified lines which are determined to not be perpendicular,collinear, or parallel to another line among the identified lines in thegradient vector image; and identifying linear objects using theremaining identified lines which have not been eliminated.
 13. Thenon-transitory computer-readable recording medium of claim 12, whereinthe wherein the operation of identifying lines using the identifiedterrain types comprises: applying the identified terrain types to thegradient vector image; determining a threshold based on mean gradientmagnitudes for each individual identified terrain type; and eliminatingpixels in the gradient vector image having a gradient magnitude notgreater than the threshold for the identified terrain type associatedwith the eliminated pixels.
 14. An apparatus for identifying linearobjects in an image, comprising: a memory configured to store the image;and a processor configured to: identify terrain types in the image;generate a gradient vector image from the image, the gradient vectorimage identifying a gradient magnitude value and a gradient directionvalue for each pixel of the image; identify lines in the gradient vectorimage using the identified terrain types in each portion of the image;determine whether the identified lines are perpendicular, collinear, orparallel to another line among the identified lines in the gradientvector image; eliminate lines among the identified lines which aredetermined to not be perpendicular, collinear, or parallel to anotherline among the identified lines in the gradient vector image; andidentify linear objects using the remaining identified lines which havenot been eliminated.
 15. The apparatus of claim 14, wherein theprocessor in identifying the lines using the identified terrain types isconfigured to: apply the identified terrain types to the gradient vectorimage; determine a threshold based on mean gradient magnitudes for eachindividual identified terrain type; and eliminate pixels in the gradientvector image having a gradient magnitude not greater than the thresholdfor the identified terrain type associated with the eliminated pixels.